June 15, 2023

What is Data Engineering? Your Path to a Promising Tech Career

7 min read

Behind every successful operation, there are vital roles performed by talented experts, and the tech industry is no exception. In a world of social media, digital marketing, and machine learning, data is the building block of success, and Data Engineers are the builders, crafting the infrastructure that many organisations rely on for their success.

That’s why Data Engineers are in such high demand and why we need more of them here in Aotearoa. Read on to find out more about this critical job, and why it’s such an exciting career opportunity for any IT enthusiast!

So, what is a Data Engineer?

The world is full of data sets, statistics, and research, but in its raw form, this data can be difficult for a business or analyst to make sense of. Or the data might exist, but no one has a way of getting at it. Enter…the Data Engineer.

Data Engineers create, maintain, and optimise the infrastructure needed to convert raw data into a usable format.

They’re the ones who access and collate data from different sources, structure the data, maintain databases and data management systems, and ensure the data provided is accurate and easily accessible. Any organisation that uses any sort of data to make business decisions can benefit from having a Data Engineer.

Take Netflix for example; how would they know which new shows to invest in, without first understanding what type of content audiences like? It all comes back to the data.

Without a Data Engineer and the infrastructure they set up, organisations end up making decisions without having the full story, and this can be detrimental to their customers – or even the organisation or indsutry as a whole – so it’s a pretty important role!

As Netflix’s Data Engineer, Kevin Wylie puts it:

“I want to make the lives of data consumers easier and enable them to be more impactful…through better access to information and insights.”

Top Tip: Connect with Data Engineers and data professionals online – places like Facebook, LinkedIn and Reddit have communities where you can learn more about their real world experiences.

Skills to pay the bills

Now you know what a Data Engineer does, let’s look at what type of skills you might need to become one:

Programming expertise – managing data sets requires understanding and practical use of programming to ensure solutions are configured properly. Some common languages that are applied in this discipline are Python and SQL.
Database design – working with various and possibly unfamiliar sources of data requires skills in handling and optimising this information. A complex or large scale database needs to be built in a way that allows rapid access to data by anyone who needs it.
API and integration skills – Data Engineers spend a lot of time working out the best way to get different systems to talk to each other. Data insights often require taking raw information from many sources and combining them.
Cloud computing – lots of data is migrated to and managed in the cloud, especially in recent years where business continuity was put to the test. A good Data Engineer knows how to get the most out of cloud systems like AWS.
Quality Control – data extracted from different systems often needs fine-tuning and some clean up before it can be reliably used by data scientists and business analysts.
Strategy – while data engineering is highly technical work , Data Engineers still need to understand, apply, and communicate the high level business impact of their efforts.

Top Tip: These are just some of the skills that are often needed, but don’t be discouraged if you don’t meet 100% of the requirements in a job ad; employers may look for soft skills and personality traits instead, and may be willing to train you in any other skill gaps!

A day in the life of a Data Engineer

A data engineer’s role is a real mixed bag. A typical day or week can involve proactive design and building of databases, or configuring data integrations, but a Data Engineer won’t always be doing ‘hands on’ work either.

You may also need to problem solve and respond to issues, as rapidly evolving tech stacks and business needs aren’t always rolled out perfectly. As any experienced IT professional would tell you, putting out a fire is par for the course!

You might be involved with one or more projects at the same time – any organisation going through ‘digital transformation’ will need Data Engineers to support the handling and connectivity of data.

You should expect to work alongside lots of different people, from software engineers and back end developers to front end designers. You could even end up supporting a senior leader. For example, you may be asked a question or presented with a strategic business goal, and tasked with figuring out how to get the best data to help provide a solution or make a decision.

Providing guidance around the implications data can have on decision-making means you’ll become strong at communicating technical knowledge with non-technical people, a skill that can unlock dozens of doors in your tech career.

Top Tip: A Data Engineer’s daily role can vary depending on the employer and project. To get a better sense of what a role would entail, make sure you read the job description carefully, then call and ask the recruiter or manager specifically what the team’s typical day is like, and what the expectations are for the role.

Career progression for Data Engineers

Like many disciplines in IT, the demand for and the still-evolving nature of data engineering means that you can find both junior and senior level roles in this space. If you’re really interested in how data is connected, accessed, and used, but only just starting out your IT career, aim to get into this space soon.

There’s no sign of New Zealand’s data needs slowing down, with many organisations still in a state of transition from traditional, manual methods into the new digital era.

There’s only going to be more roles and demand for data engineering expertise. As far as earning potential and fulfilment, this is a great field to work in.

You may have heard about Data Scientists – those who can interpret data sets and provide forecasting, modelling and other useful functions to the business. While Data Engineers can become Data Scientists, this isn’t necessarily the typical progression. Many Data Engineers are demanding equitable and often higher salaries. Again, the pay and duties will vary between organisations, so we’d suggest following the discipline you’re most passionate about.

Top Tip: Many skills you develop in IT will be highly transferable across the industry, so whether you start in data or work towards it via other roles, take time to identify the relevant skills you bring to each new role.

A better future depends on data

So many big calls are being made by organisations around New Zealand based on data. Given the shortage of data professionals across Aotearoa, the quality of data isn’t always top shelf. Often the better/worse the data served to analysts and decision makers is, the stronger/weaker the outcomes.

In this sense, a Data Engineer is fundamental to an organisation being able to make smart, informed decisions. It’s just one of many reasons why data engineering is such an exciting career path.

Whether it’s improving agricultural systems to enhance economic growth sustainably, identifying ways to better support underprivileged communities, or educating our next generation in a way that secures a happy, healthy New Zealand of tomorrow – data will power a lot of the key decisions made.

Top Tip: Data roles exist across virtually all industries now, so find one you’re passionate about!

Are you our next data hero?

If you love the complexities and detail of data but also enjoy thinking broadly about how technology and information is utilised in an organisation, then you need to explore data engineering as a potential option, whether as a long term career path or an invaluable chapter in your IT story.

Explore our available Data Engineer roles today

Check out our latest Data Engineer jobs or chat to one of our friendly team today.

Ace Your IT Interview: 10 Tips from a Recruiter

8 min read

If you want to ace your tech job interview, building rapport is key. One of the biggest challenges for professionals in interviews is how to balance professionalism while being relaxed and authentic. The more the interview feels like a natural conversation, often the better it will go. If you’re an IT professional looking to improve your interview experience, this article’s for you. We’ll explore ten ways that you can build rapport naturally to help you succeed in your next interview.

1. Email and phone conversations ahead of the interview

Building a professional relationship with a hiring manager starts before you even enter the room. There may also be others you are speaking with beyond a main manager – it’s quite common in IT for a senior developer or other technical role to be in the room as well to specifically gauge hard skills or ask technical questions.

Prior to the interview, you can use email and phone calls to build some initial rapport. This could be sharing some of your experiences, asking some questions about the team and the work, or even general reflections on the industry. Present yourself as an insightful colleague, not just a resource.

You’ll find that with some pre-meeting communication the time to break the ice in person could be reduced. There’s no need to over-do this; simply let the flow of communication translate some of your personality and not solely facts about your fit for the job in question.

2. Be honest and open

If you can remain open and honest in a job interview, you’ll have less to remember and have a smoother conversation. But it doesn’t mean you shouldn’t think about how to convey information effectively and share the most relevant things. Honesty in an interview is about being upfront with your motivations for applying, the parts of work you enjoy and don’t, and things you want to know more about. It’s also a matter of being open about who you are, not who you think the panel expects you to be – after all, they’ll learn who you are genuinely anyway if you get the role.

Any employer who wants to build or maintain a great work culture will prefer candidates be open with their skills and their personality. It takes all sorts to build a diverse, talented team, so be yourself and answer questions honestly. It’s in your best interests that the conversation is open and both parties get the most accurate sense of person-job fit. Without this, there’s the risk of getting into a role that’s actually not the best choice for you.

3. Address everyone in the interview

Many job interviews include a number of people, sometimes called a ‘panel’. While it might feel natural to just talk to one person, we’d encourage you to actively address each person. That means looking at each of the panel in the eye as you answer questions, especially those that demand a longer answer. If you get a sense of what areas of your experience are most important to each of the team, you can tailor your focus of conversation accordingly.

Addressing everyone doesn’t mean spending equal time communicating with each individual; typically, one person will lead the interview and ask the majority of questions. Some interviewers will ask their colleagues some questions. Regardless of how much involvement they have in asking you things, we’d encourage you to find ways to involve them. You can do this by asking them questions specifically, such as a technical question to a senior developer about the systems architecture. It may seem minor, but this effort will be noticed by the interviewer.

4. Don’t be afraid to share things about yourself

A job interview is about both sides finding out about the other to determine fit. With workplace culture being such an important part of managing staff attrition and team motivation, employers want to learn about you as a person beyond the skills and experience you have. While it’s important to illustrate why you’ve got the tools to thrive in a job, make sure to find opportunities to build rapport by sharing a bit more about who you are outside of work.

Not sure where to start? Some common go-tos are your family, where you grew up, went to school, past travel, your hobbies, holiday plans and even personal development. You know best what to share in the interview, but even a small conversation about yourself can help open everyone up and create a more relaxed atmosphere.

5. Ask questions about the project, team or business

You’ll likely have heard the advice of ‘asking questions’ in the interview, but it’s not simply a matter of asking any question. This is an opportunity to learn more about the job and team, as well as demonstrate genuine interest in what you’ll be doing. Before the interview, jot down 4-5 questions about the business, team or active project that you’d be helping with. Make sure these are questions that you’re actually interested in learning the answer to.

You can bring a notepad to the interview to capture notes on the answers to your questions. You might hear something that warrants follow up later via email or phone – another opportunity to show your engagement and interest in the role.

6. Be willing to adapt your responses based on what you hear

If you have limited interview experience, adapting on the fly can be challenging, but it’s a worthwhile effort for a more natural and successful interview. Preparedness is important – come with scenarios, skills, challenges and experiences you’ve acquired to help answer questions in the interview. Of course, it’s unlikely that every question will align perfectly with your readiness.

Be willing to pull from other past experiences or cover skills outside of those you’d prepared for. The conversation may also move into an area that you’ve got thoughts on but hadn’t previously considered in your preparation – be willing to provide your view, even if you need a moment to consider.

Adapting and adjusting your approach is a skill that improves over time. Remember to actively listen, and avoid relying on heavily-rehearsed responses that don’t fit the conversation, as they can hinder rapport-building with the interviewers.

7. Show you’ve done your homework

In our experience working with both employers and IT professionals, we can safely say there’s a huge benefit in candidates who’ve read up on the business they’re interviewing for. This doesn’t need to be a major undertaking either. Simply jump online and read about the organisation, check out their social media, ask around your network – easy steps that will help you build an idea of what the employer does, and their values.

Demonstrating this in the meeting can be dropped in naturally to responses and questions you ask. For example, “I noticed on the website that you work with non-for-profit clients. Was there any difference in how you engage these clients and others?”. It can be even more nuanced, such as referencing a skill or value that the employer pushes in their website or public comms and how you hold a similar value (and how you’ve applied it in the past).

Doing your homework doesn’t mean you have to memorise everything. Taking notes and bringing these in shows this homework and effort put into a good interview. You’ll notice how much this foundation of respect fosters better communication and rapport building.

8. Share views on the tech industry

The conversation doesn’t have to focus solely on the employer or yourself. Sometimes it’s valuable to share thoughts around the IT industry as a whole; emerging technologies, current trends or events or even the job market. These sorts of discussions help everyone get a sense of each other’s perspective – and set outside the dynamic of interviewer and interviewee.

9. Be honest about gaps in your knowledge – and confident in your strengths

No one’s perfect for a role. With enough digging, any IT professional will have gaps in their experience or knowledge. An IT employer’s objective is to find someone that best fits what they need. As a candidate you can build plenty of trust by being forthcoming about areas you’d like to improve in or where you’d need some training. On the flipside, don’t be shy about talking about the areas you’re particularly good at or confident in. While humility is valued in New Zealand work culture, building rapport through honesty means being open about both your strengths and areas for improvement.

10. Share your immediate reflections at the end of the interview

After the interview you can give some immediate feedback on how you thought the meeting went. This doesn’t need to be overdone – simply sharing that “I really enjoyed the conversation” is effective enough for reinforcing that you’re interested in exploring the opportunity further. If there were things you weren’t so sure about then feel free to air these. You might have simply misconstrued something that can be quickly put to bed there and then.

Learn more about New Zealand’s IT industry

If you’re interested in reading more about our tech sector, head over to the Absolute IT blog where you can find insights about a range of topics including job interviews, CVs and personal skill development.

Explore our available IT roles today

Are you looking for a new IT role, perhaps one in which you can enjoy a flexible working arrangement? Check out our latest jobs to potentially find your next big career move.

Absolute IT