Project Spotlight: Optimise Product Updates/ Creates on pipelines
Challenge - Our platform which houses millions of products were subject to regular updates by Marketplace sellers via APIs or by Kogan.com's internal teams. However, an arduous average update pipeline duration of 8 hours hindered the prompt visibility of these changes to customers on the website or app.
Determined to tackle this challenge, our focus rested on expediting the update process while concurrently reducing infrastructure costs so that our customers can swiftly access the most recent product information.
An analysis of the update pipeline revealed several areas requiring our immediate attention:
Repetitive Update Events - recurrent API calls made by Marketplace sellers resulted in duplicated update events for the same product. This redundancy significantly impeded operational efficiency. Sluggish Database Queries - inefficiencies in database queries engendered prolonged processing times, undermining the timely dissemination of updated product information. Architectural Bottlenecks - restrictive architectural elements posed as bottlenecks, impinging on system throughput and curtailing overall performance.
To overcome these hurdles, the following measures were taken:
Enhanced Diffing Logic - by leveraging an opensearch cluster, we revolutionized our diffing logic. This sophisticated cluster facilitated real-time comparison of incoming API payloads. Consequently, if no alterations were detected in the product information, redundant requests were skipped, culminating in a more expeditious system. Optimized Database Queries - a Thorough optimization of database queries was undertaken, incorporating prefetching techniques and mitigating the prevalence of joins and n+1 queries. This meticulous overhaul addressed the root causes of sluggishness, resulting in expedited processing times. Introduction of a Datastore - a revolutionary datastore was introduced, specifically designed to house denormalized product data. This datastore formed the bedrock of product information display to customers, effectively reducing dependence on Opensearch and enabling its gradual scaling down. The introduction of this refined infrastructure bolstered system responsiveness and agility.
By implementing these measures, we were able to address inefficiencies on our product updates in a timely manner. Our actions played a big part in providing our customers a more seamless experience on our site! This also enables us to scale our operations so we can support a wider range of products and continue meeting the needs of our customers in the future.