Four Years Strong: Celebrating Our Koganniversaries

In a talent landscape full of competitive opportunities, where change and turnover are part of the norm, Kogan.com stands out as a place where people choose to stay, grow, and advance their careers. Many of our team members have long tenures, with some contributing as long as 12 or 15 years, reflecting the strong culture and opportunities here.

This month, we celebrated three team members reaching their four-year milestone. To mark the occasion, we spoke with them about what has made their journey so rewarding, what has kept them at Kogan.com, and what continues to inspire and excite them as part of our team.

Adam Slomoi

Adam is currently the Tech Lead of a squad. He joined Kogan as a Software Engineer and quickly progressed to Senior Software Engineer, and most recently to Tech Lead. In this role, he continues to sharpen his technical skills while growing his passion for people leadership.

How has your role or perspective on engineering evolved over the last four years?I’ve been fortunate to work across different areas of the business and on various components of our system during my time at Kogan. One theme that has remained consistent throughout, and that I’ve gained a greater appreciation for, is the focus on business outcomes.

How did you feel stepping into your first leadership role, and what did you learn from the experience?
I felt well prepared before officially taking on a leadership role. We have a very collaborative team, and there have been many opportunities along the way to have a say and help set the direction for the team.

Sam O’Halloran

Sam started at Kogan in his first formal software engineering role straight out of university. He has honed his technical skills, contributed significantly to his team, and grown in both technical depth and breadth, applying best practices efficiently.

Which project are you most proud of, and what made it exciting or unique?
I’m most proud of the work on product variants. It wasn’t one big launch, just lots of smaller fixes that added up. I tightened grouping logic in pipelines and batch jobs, cleaned up SPS edge cases, and ensured changes flowed through to OpenSearch. On the UI side, I tweaked filters, added a simple horizontal selector for single-variant products, enabled a promo palette, fixed dropdown ordering, and added a VariantGroup sitemap for better SEO. It was satisfying because variants are messy in the real world, and these changes made choosing the right option faster and clearer for customers, and saner for our internal teams.

Can you share a moment where your work made a noticeable impact for the team or the product?
I led a performance pass on our Product Listing Page. I cut work above the fold, switched tiles to responsive images, added light skeletons, virtualised the brand filter with React Virtuoso, lazy-rendered heavier lists, and fixed scroll and CLS issues. The page now loads faster and filtering feels smoother, which reduced the performance issues we were tracking on that screen.

I also flagged follow-ups for our Product List endpoint to make it simpler, faster, and more reliable.

Yanxu Zheng

Yanxu is a highly skilled Senior Software Engineer who has developed deep technical expertise. He was recently promoted to a tech lead role and is now responsible for leading a large squad.

What’s one of the most interesting technical challenges you’ve solved during your time here, and how did you approach it?
Our keyword-based search sometimes failed to grasp user intent, leading to irrelevant results. I tackled this by leading an experiment to test whether a hybrid search model combining keyword and semantic matching could perform better. To avoid any risk to live services, I ran the experiment on a parallel cluster that mirrored production data. We then conducted an A/B test comparing the new hybrid search against the existing keyword search.

The results were clear: while the new model showed a modest improvement in relevance for some queries, it produced no statistically significant lift in user conversion. Based on the data, we decided not to roll out the feature, as the additional infrastructure cost wasn’t justified by the lack of business impact. My key takeaway was that a technical improvement is only valuable if it moves a core business metric. This experience reinforced my approach of always tying engineering efforts to measurable business outcomes.

Is there a problem you tackled that taught you a new skill or changed how you think about engineering?
I once fixed major performance issues in our OpenSearch cluster by challenging the official best practices. The standard advice was to use 10–50GB shards, but our search latency was poor. I ran experiments and proved that for our specific workload, smaller 5–10GB shards were far more efficient. Switching to smaller shards cut our query latency by over 20%. The experience taught me to always validate standard guidelines with real-world data, as optimal solutions are context-dependent.

Building Your Own AI Agent

The DEBI (Data Engineering and Business Intelligence) team recently attended the DataEngBytes 2025 conference, where the hot topic for the year was, unsurprisingly, AI agents. My favorite talk, by Geoffrey Huntley, presented a powerful and surprisingly simple idea: It’s not that hard to build an agent; it’s a few hundred lines of mainly boilerplate code running in a loop with LLM tokens. That’s all it is!

Kogan DEBI team at the DataEngBytes conference 2025

The speaker’s main point was that things were developing extremely rapidly in the AI space, but rather than worrying about how AI might take engineering jobs in the near future, we should become AI producers, leveraging agentic AI to automate things, from data pipelines to our own job functions. Understanding this is, in his words, "perhaps some of the best personal development you can do this year."

This idea is both liberating and empowering. It transforms the conversation from one of anxiety about job security to one of excitement about a new, fundamental skill. Let's pull back the hood on how these agents work and understand the simple primitives that allow us to become producers of automation, not just consumers.

The Fundamentals: The Shift from a Tool to a System

Before you write any code, you need to understand the new paradigm. We’re moving beyond just using Generative AI (Gen AI) as a tool and are now using it to build a complete system: an AI Agent.

Generative AI (Gen AI): The Creator: This is the broad category of AI models that are designed to create new content. LLMs are the most common form of this. They are reactive; you give them a prompt, and they generate a response—be it text, code, or an image. Gen AI is the creative engine.

Agentic AI: The Doer: This is a type of AI system that is designed to act with autonomy. You give it a high-level goal, and it uses its "brain" (a Gen AI model) to reason, plan, and execute actions to achieve that goal. This is the proactive part of AI. The speaker referred to these as "digital squirrels" because they are biased toward action and tool-calling, focusing on incrementally achieving a goal rather than spending a long time on a single thought.

The key insight is that the most powerful Agentic AI systems use a highly-agentic Gen AI model (like Claude Sonnet or Kimi K2) as their core decision-maker, and then wire in other, more precise Gen AI models (the "Oracles") as tools for specific tasks, like research or summarization.

The Agent's Heartbeat: The Inferencing Loop

An agent's core function is an elegant, continuous loop. It's the same loop that powers every AI chat application, but with one critical addition: the ability to execute tools.

1. User Input: The agent takes a prompt from the user.
2. Inference: It sends the prompt, along with the entire conversation history, to the LLM.
3. Response Analysis: It receives a response. If the response is a direct answer, it's printed to the user.
4. Tool Execution: If the response is a "tool use" signal, the agent interrupts the conversation, executes the specified tool (a local function), and then sends the result back to the LLM to continue the conversation.

This simple, self-correcting loop is the engine that drives all agentic behavior.

The Building Blocks: Primitives of a Coding Agent

The power of an agent comes from its tools. A tool is simply a local function with a description, or "billboard," that you register with the LLM. The LLM's training nudges it to call these functions when it believes they are relevant to the user's request. The workshop demonstrates five fundamental tools for building a coding agent.

• Read Tool: This tool reads the contents of a file into the context window. It's the first primitive, allowing the agent to analyze existing files.

• List Tool: This tool lists files and directories, giving the agent awareness of its environment, much like an engineer running ls to get their bearings.

• Bash Tool: This powerful tool allows the agent to execute shell commands, enabling it to run scripts, check processes, or interact with the system. It's the key to making the agent's work actionable.

• Edit Tool: This tool allows the agent to modify files by finding and replacing strings or creating new files. When combined with the other tools, it completes the agent's ability to act on the codebase.

• Search Tool: This tool uses an underlying command-line tool like ripgrep to search the codebase for specific patterns. It helps the agent quickly find relevant code without having to read every file.

Putting It All Together: The FizzBuzz Example

By combining these primitives, an agent can perform complex, multi-step tasks. In his talk, Geoffrey illustrated this by having an agent solve the classic programming problem of FizzBuzz. This is a classic programming exercise that requires a program to print numbers from 1 to 100, with a few simple exceptions: for multiples of three, print "Fizz" instead of the number; for multiples of five, print "Buzz"; and for numbers that are multiples of both three and five, print "FizzBuzz."

By giving the agent the prompt, "Hey Claude, create fizzbuzz.js that I can run with Nodejs and has fizzbuzz in it and executes it," we are asking it to orchestrate a multi-step process. The agent will use the Edit Tool to create the file and then the Bash Tool to execute the script and verify the output. The speaker then took it a step further, asking the agent to amend the code to only run to a specific number. The agent successfully handled this by using the Read Tool to check the existing code, the Edit Tool to change it, and the Bash Tool again to verify the new output. This ability to continuously loop back on itself to correct and refine its work is the key to a true agentic system.

The Career Implications for Engineers

In the last six months, AI has become "incredibly real," and the ability to build these systems lets us become producers of automation, not just consumers of it. And here's the best part: the skills are completely transferable. The same principles used for a code-editing agent can be applied to automating data pipelines, CI/CD workflows, database management or even parts of our core job functions.

The final message from the talk was super clear: this technology is here, and it’s surprisingly accessible. As engineers, our value in the coming years is going to be defined by our ability to use and produce automation. The most successful Engineers aren't the ones who fear AI, but the ones who embrace it and learn to build these powerful tools. There’s nothing mystical about agents; they're just an elegant loop built on a few core principles. The next step is to start building one yourself.

Further Resources

https://ghuntley.com/agent/ https://ampcode.com/how-to-build-an-agent

Order Dispatch Systems at Scale

How to load-balance like a seasoned waiter

Software systems often parallel the real world. Imagine running a busy restaurant, where customers line up to make orders whilst the kitchen prepares the meals. In the software world, your users are the customers, and your backend services are the kitchen. With more people online than ever before, that line might start to grow out the front door. The ability to scale is no longer optional, it is essential.

Know Your Options

Vertical Scaling - Scale Up	Horizontal Scaling - Scale Out
Expanding your restaurant by adding more tables or a larger kitchen. In software terms, this means scaling up your infrastructure. More powerful CPUs, larger memory, increased throughput etc. This is a relative quick fix, but comes with diminishing returns and limits on how big everything can get.	Opening new restaurant locations to serve more customers simultaneously and distribute existing flows. In software terms, this means adding more API servers, more worker nodes or creating many database replicas. This approach is more flexible and scalable than vertical scaling in the long term.

A Steppingstone - Queues

Just like how customers queue for their order, we create a pull-based task queue for our order management system using AWS Simple Queue Service (SQS). Tasks get queued into the SQS, and a consumer service will continuously poll this queue to process the tasks.

This gives a lot of control for the queue consumer to dictate the frequency of polling, which works well in systems that cannot handle high throughput or requires non-concurrency like the SAP ERP (more on that later). SQS also provides built-in dead-letter-queues, retry policies, at least once delivery guarantee and scales automatically.

Vertical scaling involves sizing up the compute power of the consumer (CPU, RAM etc). Horizontal scaling involves spinning up more consumers of the SQS.

However, queues have limitations:

• Latency between order arrival and processing. 

• Inefficient polling process that checks constantly even when the queue is empty.

• Limited fan out, message is designed to be consumed by one service making it difficult for different components to react to the same event.

• Concurrency issues, need to tune the message visibility timeout to ensure it isn’t picked up by another consumer instance when scaling out.

The Gold Standard - Events

Scalability starts at the architectural level; enter event-driven programming. Instead of queuing up, customers scan a QR code, and their order is sent instantly to the kitchen, the waiter, the pay desk all at once. No delay, no queues.

We recreate this by having an event publisher that sends messages to an event bus. Which notifies the event subscribers: warehouse, emailing service and SAP simultaneously, allowing them to react to the same order independently. Adding the previous vertical and horizontal scaling options mentioned above, creates a powerful system for processing and dispatching orders as the separate components can be scaled independently. Which also lends well to a microservices architecture.

This model mitigates a lot of the previous approach’s deficiencies:

• Lower latency, as it is push based not pull based.

• Subscribers can be scaled individually.

• Event bus is built to handle concurrency.

There are two ways to implement this in AWS: EventBridge and SNS (Simple Notification Service). We choose EventBridge for its ability to handle more complex workflows and native integrations with third party SaaS applications like Zendesk. 

Unlike the SNS approach where messages must be published to a specific topic and risks the number of topics growing too large; EventBridge receives from many sources at once. With advanced filtering capabilities, it can inspect the full event payload and route them to the appropriate consumers. Additionally, event archiving and replays are also supported for improved debugging. 

Here is the basic implementation:

• Publish events to the EventBridge from your application

• Define Event Rules that filter events based on its payload

• Configure Targets for each rule - they can have multiple targets

Our target will be SQS as it allows our preexisting .NET services to plug into the new event-driven system without major modifications. However, serverless lambda functions are on the table if we can remove the dependency on SAP, more on this later.

While powerful, event-driven architecture is not without its drawbacks:

Debugging and tracing: Events are asynchronous and loosely coupled making it difficult to find cause and effect. Need to set up comprehensive logging and distributed tracing. 

Eventual consistency: System components may be temporarily out of sync. Making it harder to understand behaviour. Logic must also be built to handle stale data gracefully.

Event schema evolution: Changing the payload structure can cause breaking impacts for downstream services. Need to document clearly how the payload is consumed and have a versioning strategy.

Despite these challenges, with the right tooling and implementation, event-driven architectures can be made highly observable, testable and resilient. 

A Slow Chef in the Kitchen

Sometimes, the bottleneck isn’t your system but it’s external dependencies. In our case, SAP is that slow chef. The Data Interface API is single threaded and does not support batch processing. If you throw too many requests, it will choke no matter how fast other components are. Identification of such bottlenecks are crucial, lest those other scaling efforts are wasted. 

Luckily, in our case we can upgrade SAP from the DI API to a modern alternative called the Service Layer. It is designed with scalability in mind:

• Uses HTTP and OData protocols

• Can parallel process

• Automatic load-balancing

• Does not require local installation like DI API

These properties make it much easier to develop web and mobile applications which are more accessible than the SAP Windows client. The service layer’s more stateless nature lends nicely to the aforementioned event-driven architecture, bringing SAP in line with the rest of our scalable system.

Conclusion

Just like a restaurant, software systems should be designed with future scalability in mind. Start with simple abstractions like task queues and evolve to fully decoupled event-driven systems. Horizontal scaling is often better and more flexible than vertical scaling. When faced with external bottlenecks, tackle them head-on. Architecture is the business, with intentional design, your restaurant won’t just keep up but thrive.

References

1. The Complete Guide to Event-Driven Architecture | by Seetharamugn | Medium

2. The Art of Scaling: Building Systems for Millions of Users | by Alex Glushenkov | Medium

3. Amazon SQS Features | Message Queuing Service | AWS

4. Amazon SQS, Amazon SNS, or Amazon EventBridge? - Amazon SQS, Amazon SNS, or EventBridge?

5. Service Layer 1/6: API

6. A Guide to Event-Driven Architecture Pros and Cons | Solace

Empowering Data Through Self-Service: Behind the Scenes of Our Data Platform

At Kogan.com, our data needs have grown alongside the business. As more teams relied on insights to move quickly, it became clear our request-based BI model couldn’t scale. We needed a platform that empowered teams to answer their own questions, trust the numbers, and move independently. That journey led us to build a self-service platform grounded in governance, transparency, and scalability—powered by dbt, Looker, and Acryl (DataHub).

Rethinking Our BI Model

We originally relied on Tableau. It served us well but had limitations: duplicated logic, inconsistent metrics, and limited collaboration with dbt. Tableau workbooks weren’t version-controlled, which made maintaining consistency difficult. To bridge modeling and reporting, we often created extra presentation tables in dbt, adding complexity. We needed a platform that integrated tightly with dbt and supported governed exploration.

A New Architecture: Modular, Transparent, Scalable

We redesigned the platform around a clean, modular flow: Raw Sources → BigQuery → dbt → Looker → Acryl (DataHub) Our data transformations are built in dbt, where we follow a layered modeling structure. While we use stg_ (staging) and int_ (intermediate) models primarily for data cleaning and standardization, the marts_ models are the ones that power our analysis and reporting. These models contain our fact and dimension tables, fully aligned with business logic and ready for consumption in Looker. We’ve integrated CI/CD pipelines using GitHub Actions, and every change is tested before deployment. This includes dbt tests, schema validations, and model documentation to ensure confidence at every layer.

Why Looker Was the Right Fit for Self-Service

Looker offered a structured, governed approach that aligned with our dbt-first architecture. LookML let us centralize business logic, version it with Git, and deploy changes through CI/CD. With support for multiple environments (UAT and Production), we can test safely before releasing to users. The Explore interface gives business users guided access to curated datasets—no SQL required. Users can drill down, apply filters, and explore confidently. This was a big shift from the Tableau model, which often required analyst support. Looker also includes row-level security, role-based access, and an AI assistant that supports natural language queries and chart generation—lowering the barrier for non-technical users. We’ve also developed internal dashboard standards—consistent layouts, filters, and naming conventions—to ensure usability and reduce support needs.

Bridging the Migration

We’re currently in the process of migrating our Tableau reports into Looker. While Looker is more efficient to build with, the migration isn’t just about re-creating dashboards—we’re using it as an opportunity to improve them. For each report we migrate, we review and sometimes refactor the associated dbt models to ensure the logic is clean, reusable, and well-documented. We also take time to redesign visual layouts to be more intuitive and self-service friendly—adding better filters, descriptive labels, and drill-down paths wherever we can. It’s not just a tech migration—it’s a platform and user experience upgrade.

Data Discovery and Observability with Acryl

Alongside dbt and Looker, Acryl (DataHub) has become the foundation of our metadata ecosystem. Acryl helps both technical and non-technical users understand what data exists, where it comes from, and who owns it. It provides searchable documentation, field descriptions, ownership metadata, and lineage tracing across dbt, BigQuery, and Looker. We also rely on Acryl’s observability features for monitoring anomalies and surfacing potential data quality issues. While it's not a test framework like dbt or a freshness tracker, Acryl helps us detect behavioral anomalies, unexpected changes, and broken relationships before they impact end users. Acryl's AI-powered documentation suggestions have also saved us time when onboarding new models or enhancing existing ones, especially for adding descriptions and tags at scale.

Lessons We’ve Learned

If there’s one takeaway, it’s that data tools need infrastructure—both technical and human. You can’t just launch Looker and expect adoption. You need a warehouse that reflects the business, models that users can trust, documentation that’s visible, and governance that feels supportive, not restrictive. We also learned that governance isn’t about locking things down—it’s about making things clear. When users understand what data means, how it’s calculated, and who owns it, they feel empowered, not limited.

Final Thoughts

We didn’t build a self-service platform just to save time—we built it to build trust. By aligning tools like dbt, Looker, and Acryl into a unified ecosystem, we’ve created something bigger than a data stack. We’ve created a culture where teams are empowered to explore, ask better questions, and make faster decisions—without sacrificing governance or quality. This transformation didn’t happen in a vacuum. It was made possible by the incredible efforts of my team—engineering, analytics, and enablement working hand-in-hand. The commitment to transparency, maintainability, and user empowerment is what brought this platform to life. We’re still learning. But we’re proud of how far we’ve come—and even more excited about where we’re going.

Beyond the Code: Threat Modeling as Your Security Superpower

As developers, we pour our energy into building robust, elegant software. We craft features, optimise performance, and squash bugs. But in today's world, building secure software is just as crucial. Enter Threat Modeling – not as a bureaucratic chore, but as a practical superpower for developers aiming to build resilient applications.

Think of threat modeling as structured foresight: anticipating how things could go wrong from a security perspective before they happen. It’s about stepping into an attacker's mindset to find weaknesses in your own designs. This proactive approach helps weave defenses right into your application's fabric from the start.

Understanding the Battlefield: Core Security Lingo

To talk effectively about security, we need shared terms. Start with your Assets – the valuable parts of your system, like user data or API keys. These face potential Threats, specific actions that could cause harm, often initiated by Threat Agents like hackers or malware.

To guard against threats, we use Controls (or Countermeasures). These are your front-line defenses designed to prevent attacks or detect them early, like authentication checks or input validation.

But what if a threat gets through? That's where Mitigations come in. These are measures aimed at reducing the damage if a control fails and an attack succeeds. For example, while access controls prevent database intrusion (a Control), encrypting the data reduces the impact if someone does get in (a Mitigation).

Finally, be aware of Trust Boundaries – the lines separating parts of your system with different security levels (like frontend vs. backend). Interactions across these boundaries need special attention.

This common language helps us pinpoint and discuss security risks clearly.

Why Add Threat Modeling to Your Toolkit?

"Another process?" you might ask. But integrating threat modeling saves time and headaches later. Catching a security flaw during design is far cheaper and easier than patching a live system under duress. It helps you avoid building inherently vulnerable features or adding complexity that inadvertently creates attack surfaces. Plus, it fosters a shared understanding of architecture and risks across the team. While ideal early on, threat modeling adds value at any stage.

Walking Through the Process: A Practical Framework

A helpful way to approach threat modeling is by systematically answering a few key questions, often broken into phases:

Phase 1: Mapping Your Territory (What are we working on?)

First, define your scope clearly. What system or feature are you analysing now? What's explicitly out of scope? Identify the critical Assets within these boundaries needing protection. Then, break the system down into its core components – external entities, internal processes, data flows, data stores (databases, caches), and map out the Trust Boundaries.

Visualising this is key. Creating a Data Flow Diagram (DFD) using tools like Draw.io or Miro provides an invaluable map. This diagram isn't just for the threat modeling session; it becomes useful system documentation and helps everyone visualise potential weak points.

Phase 2: Anticipating Attacks (What could go wrong?)

With your map, brainstorm potential threats. Put on your "attacker hat." How could someone misuse this system? The STRIDE framework is excellent for this:

• Spoofing: Faking identity (user, server).
• Tampering: Unauthorised modification of data or code.
• Repudiation: Denying an action performed (aided by poor logging).
• Information Disclosure: Leaking sensitive data.
• Denial of Service (DoS): Making the system unavailable.
• Elevation of Privilege: Gaining unauthorised higher-level access.

Examine your DFD, focusing on data crossing trust boundaries, user inputs, and external interactions. Ask, "How could STRIDE apply here?" Go through each category for relevant parts of your system to uncover potential vulnerabilities.

Phase 3: Building Your Defenses (What are we going to do about it?)

Identifying threats is only half the job. Now, decide how to respond. For each significant threat, consider your options. Can you implement Controls (like strong auth, input sanitisation, encryption) to prevent or detect it? Perhaps you can Eliminate the threat by redesigning a workflow or removing a component. Sometimes, you might Transfer the risk (e.g., using a third-party payment processor). For low-probability, low-impact threats, you might consciously Accept the risk, documenting the decision. Prioritise countermeasures for the highest-risk threats first.

Phase 4: Staying Vigilant (Did we do a good enough job?)

Threat modeling isn't a one-off task. Systems evolve, new features appear, and attackers adapt. Therefore, it's crucial to Review and Refine your threat models periodically, especially after significant changes. Keep diagrams and threat lists current. If you handle sensitive or regulated data (GDPR, PCI), ensure your model addresses Compliance needs. The goal is to Integrate threat modeling thinking into your regular development rhythm.

Level Up Your Security Posture

Threat modeling empowers developers to build security into applications proactively. By considering potential attacks and designing defenses early, you create more resilient, trustworthy software, saving significant time and resources compared to reacting to incidents. Start small, pick a critical feature, map it out, think through STRIDE, and make security an integral part of your development craft.

Have fun!

*This article was written with AI assistance :)

A New Software Engineer’s Journey at Kogan.com

We are always excited to welcome new talent to our team! At the heart of our engineering culture is a robust and agile onboarding process that gets new joiners up to speed in no time. We pride ourselves on helping our teams hit the ground running, shipping fast, experimenting boldly, and embracing failure as a valuable part of learning.

Our onboarding is designed to immerse new team members in the action right away. From day one, they’re trusted to roll up their sleeves and tackle exciting challenges. Teams are encouraged to ship their first changes early, iterating quickly and gaining exposure to the "fun and gritty" aspects of the work. As part of our dev blog series, we speak to Fraser, one of our newest engineers, to hear about his experience joining the team and how our unique approach has supported his journey so far.

What was your first impression of the team and the kind of work you get to do here?

Joining this team has been an incredible journey, working alongside highly experienced engineers has been inspiring. It’s a fantastic environment to grow, learn, and problem solve alongside some of the best minds.

Can you describe the first project or task you worked on after joining?

I’m currently working on a project with one of Kogan.com’s verticals. It's a migration and integration project with a major financial institution. It’s been an exciting new experience, offering valuable insights and opportunities to learn.

What excites you most about the technology or product you're contributing to?

I hadn’t worked with the Django framework commercially before, but it’s been exciting to explore the structure of a large-scale Django application and come up with innovative ways to future-proof it.

How does this experience compare to other roles or teams you've been part of?

Kogan.com’s focus on in-person collaboration is incredibly refreshing. I truly value the emphasis on face-to-face conversations and the ease of tapping someone on the shoulder to quickly understand different parts of the application. It’s absolutely invaluable 🙂

Making Smarter A/B Testing Decisions with Event Tracking and Session Replays

A/B tests are theoretically simple but sometimes offer challenges when insufficient data or events are tracked. In e-commerce, conversion rates are often highlighted as a key metric but what causes it and why. By leveraging additional tools that enable event tracking and session replays such as FullStory we can attach context and understand what users are actually doing, allowing us to make data-driven decisions which is crucial in modern business.

Event Tracking: What Happened

When it comes to e-commerce, every click matters, which is why we need to have an event-tracking mechanism. Event-tracking enables tracking user actions across your website, such as adding an item to the cart, hitting a call to action button, or proceeding to checkout. This data is essential in understanding the different behaviors that exist among your A/B test variants.

For example, if you’re testing two versions of a product page, event tracking helps you see:

• Click Rate: Which of the two gets more clicks on the ‘Add to Cart’ button?
• Engagement: How long do users spend on the page and how many elements such as images, product descriptions, or reviews do they interact with?
• Form Submissions: What user feedback method leads to the best uptake for optional benefits?

This approach is beneficial because it allows you to understand what’s happening behind the scenes, beyond just looking at the final sales numbers.

Session Replays: Why it Happened

While event tracking shows what happened, session replays reveal the why. Watching a replay of a customer’s experience (with sensitive data masked) often uncovers behaviors and friction points you, as the developer, didn’t anticipate or encounter during testing. It’s an insight you simply can’t get from final sales numbers, and it’s invaluable when trying to identify behavioral patterns or usability issues.

For example, if event tracking shows a significant drop-off with Variant A users who aren’t reaching the checkout page, session replays might reveal the layout is confusing or that error messages aren’t clear enough.

Here are some of the key benefits:

• See Drop-Off Points: Identify the exact actions that lead to user drop-offs. Are customers struggling to find important information? Are they hesitating after reading a specific detail or policy that prevents them from moving forward?
• Spot UX Issues: Observe how users interact with different elements and locate potential friction points. Is adding items to the cart not as intuitive as you thought?
• Analyse Navigation Patterns: Understand how users move through your site, and see if anything disrupts their journey. Are there too many steps to complete a purchase? How many clicks does it take before users finish their session? Does your marketing align with their needs?

Making Data-Driven Decisions

The key to making smarter decisions from A/B tests is combining event tracking data over a sufficient period with the visual insights gained from session replays. Some changes show clear trends in just a few days, but for more subtle tweaks, you may need weeks or even months to avoid making decisions based on daily fluctuations. Here’s the typical process:

• Implement Variants: Roll out the new feature to 50% of users while keeping the original version as a baseline. It's best to test one factor at a time, but you can run multiple variants if you’re working with short development cycles, as long as you have a clear understanding of the baseline performance.
• Gather Data: Track key metrics like clicks and form submissions to gauge how users interact with each variant. In e-commerce, the conversion rate is often the most important data point, but depending on your test, other interactions may be more insightful.
• Watch Replays: If the data shows significant changes or unexpected differences, watch session replays to uncover the reasons behind user behavior. This helps you reach the root of issues that numbers alone can't explain.
• Prioritise Changes: Use the data and replays to decide what improvements will have the biggest impact. It could be a simple design tweak, a clearer call-to-action, or even a complete layout overhaul. The key is to be data-driven, avoiding assumptions or personal biases.
• Iterate Quickly: Roll out small, incremental changes, monitor the results, and keep iterating. A continuous testing and improvement cycle is essential for long-term success.

Wrapping It Up

While conducting an A/B test it's essential to combine event tracking with session replays. Event tracking shows what users are doing, while session replays reveal why they behave that way. Together they form the required knowledge for developing data-driven strategies, addressing user problems, and enhancing the quality of the product. As developers this allows us to move quickly, and most importantly, to align with the business strategy.

Kogan.com Engineering Growth Paths: From Pricing Manager to Data Engineer

Committed to learning and continuous improvement, Kogan.com’s Engineering team develops its engineering talent through giving and taking responsibility, co-creation, mentorship, and internal mobility opportunities to grow and advance their careers. There are opportunities for Engineers at Kogan.com regardless of background. Some engineers at Kogan.com are Individual Contributors, Tech Leads or People Managers – and engineering growth paths and aspirations are supported throughout their journey. Featured here is Reuben Orange, our latest addition to the team who joined us through Kogan.com’s internal mobility program. After a highly successful 10-year journey in the Purchasing team, we were excited to support Reuben's career aspirations and his passion in all things data and software engineering.

With an Educational background in Mathematics and extensive experience across various roles within Purchasing, Reuben brings a unique skill set to his new role.

Collaborating closely with our Data Engineering and Business Intelligence squad,Reueben plays a crucial role in developing, managing, and optimizing infrastructure, tools, and processes important for meeting Kogan.com's analytics and data requirements.

Reuben's wealth of domain experience, coupled with his genuine passion for data and meticulous attention to detail, positions him as an outstanding Data Engineer member to the team. Tell us Reuben….

What initially sparked your interest in transitioning your previous role as pricing manager to data engineering? The pricing manager role was created to develop a “pricing strategy”, I called it “making sure we don't end up on an episode of Hoarders”. We needed to bring Kogan’s inventory level down from our very high post COVID levels, to a more reasonable position, while salvaging as much value as possible. Together, the whole team did that very successfully, and we now get much more value from a dollar invested into inventory than we did before. The role involved a lot of data querying, cleaning and crunching, which I enjoyed. And now as a data engineer, I can do even more of that! While also learning from the amazing DEBI (Data Engineering and Business Intelligence) team.

What were the biggest challenges you faced during your transition, and how did you overcome them? Wrestling with new tools has been a challenge, pushing, pulling, sprinting. I was comfortable living in a spreadsheet, but now I live in the belly of a python script. But I reckon the biggest challenge has been letting go of the old role. Old tasks die hard!

Can you share some specific examples of how skills from your previous role have been valuable in your new role? I am lucky in my old role I gained experience writing SQL and building dashboards, with the help from some great mentors. But a large part of the data engineering role is playing detective, you have to understand the data, and how it’s all connected, to get some value from it. So it has helped having experience as an end user of the data. For example, how are the different objects in the admin panel connected, or what do we mean when using different terms like AGPDI or Gross Sales.

How has your day-to-day work as a data engineer differed from your previous role as a pricing analyst? I would say the biggest change is the Engineering way of working, agile. Every day, we have a stand up to chat about what we’re working on, and what we’re planning to work on, and if anything is blocking us. When completing a piece of work, we always get feedback on it from the rest of the team, everybody knows a lot about what I’m working on, and could easily pick up where I left off. We also have continuous improvement baked in, with fortnightly retrospectives, where we look back at what did and didn’t work.

What advice would you give to others looking to make a similar career transition? I wouldn’t say it’s easy to get into software engineering, it's a lot of work, but I would say it’s very accessible, more so than ever. There is a rich vein of golden knowledge out there on the internet, you just have to mine it.