Several people have asked me about my work at JD.com. This page describes some of the products my team developed and the technical research that was developed as part of this work. During 2017-2019, I was on leave from Stanford and worked in JD’s Silicon Valley office as Chief Scientist for Business Strategy and afterward was affiliated with JD’s Intelligent Ads Group (2019-2021).
JD.com Overview (2019-2021 timeframe)
JD.com is China’s second largest e-commerce company. JD’s 2020 revenues of USD 114.3B made it China’s largest internet company by revenue: see here for more facts about the company. It is also one of China's largest Retail Media Network offering advertising on its e-commerce and content platform and on third-party inventory via its Demand Side Platform (DSP).
Broad Focus
My role was focused on using marketing science and applied econometrics to drive growth for JD and its partner brands in China, leveraging JD’s data assets and AI-driven technology platform and its large Retail Media Network. This article describes at a high level some of the goals of the broad effort: https://finance.sina.cn/2018-10-18/detail-ifxeuwws5653433.d.html (translation required). See here and here for some broad overviews of the JD Retail Media Network (Marketing 360).
Together with Paul Yan, President of JD Business Growth, engineering leaders Jack Lin and Lei Wu, and our incredible data science and economics team, I developed products related to e-commerce marketing, pricing and advertising for improved brand-building, monetization and ad-strategy for JD and participating brands.
A few specific products developed for the Retail Media platform are described below. These products are deployed on "JZT" (https://jzt.jd.com/), JD’s ad campaign management platform.
This effort involved developing a data-driven attribution framework that can facilitate reporting, bidding and budget allocation on JD's Retail Media Network. Together, the idea behind these products is for advertisers and the ad-platform to understand which touch-points are performing most effectively in driving the advertiser's campaign goals, which in turn facilitates better bidding and budget allocation via AI-driven automation.
JD Multi Touch Attribution (MTA) develops a data driven attribution product to drive attribution reporting and bidding automation for advertisers on JD's Retail Media Network. It uses a Recurrent Neural Network trained on granular user data along with Shapley Values to develop a data-driven attribution system to assess ad campaign performance. A related product, “Path to Purchase,” visualizes consumer click-steams so as to provide more data-driven context to the reported attribution.
Touchpoint Mix Modeling (TMM) develops an advertiser-facing product for campaign-level automated bidding and budget optimization tailored for ads bought via Real Time Bidding (RTB). The system integrates with modern MTA systems to provide bid and budget optimization taking attribution from MTA models as an input. A premium is placed on transparency and interpretability of the model to preserve advertiser trust in the automation.
Overview of algorithm, deployment and randomized controlled evaluation: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3913039
This effort involved developing an advertiser-facing ad-experimentation framework for JD's Retail Media Network. It comprises a set of products that together facilitate scalable, self-serve experimentation by advertisers to evaluate and enhance their advertising campaigns. Together, the idea behind these products is to provide causal lift measurement and testing to advertisers so as to improve their ability to run better campaigns in the JD Retail Media Network.
JD Conversion Lift for RTB develops a system for measuring ad-lifts in a Real Time Bidding environment. The product provides causal measurement of campaign performance to JD's advertisers. A new aspect of the framework developed is it facilitates simultaneous parallel experimentation across multiple advertisers and campaigns in such an environment, taking special care to carefully define and articulate the interactions that occur when experimentation occurs in parallel.
Overview: https://arxiv.org/abs/1903.11198.
JD Conversion Lift for AdX develops a system for measuring ad-lift in an real-time bidding (RTB) environment from a Demand Side Platform (DSP’s) perspective. The main problem is as follows: the DSP, of which JD is an example, submits bids on behalf of advertisers into external ad-exchanges (AdX). The DSP cannot control the AdX’s auction or observe its auction-queue, so typical experimentation strategies such as randomizing users into ad-exposure or not, is either infeasible or impractical. Instead, this framework facilitates experimentation by randomizing the bids submitted by the DSP into the AdX auction. The randomization is implemented adaptively, via a contextual bandit, so that the optimal bidding policy is also learned alongside inference of the ad-lift, so that the algorithm simultaneously implements lift measurement and optimal bidding, while saving on experimentation costs.
Overview: https://arxiv.org/abs/1908.08600.
JD Comparison Lift is an "A/B" testing platform for JD's advertisers to optimize their campaigns. It enables an advertiser designing a campaign to discover via online experimentation, a creative-target audience combination that provides her the highest expected payoff. The target audiences can be complex, potentially overlapping with each other, and the creatives can be any type of media (picture, video, text etc.). The algorithm is set up as a contextual bandit that adaptively allocates traffic during the test so as to minimize the cost to the advertiser from experimentation. Broad overviews here and here.
Overview of algorithm is here: https://arxiv.org/abs/1907.02178.
Overview of product development, deployment, engineering and business impact is here: https://arxiv.org/abs/2009.07899.
More advanced bandit algorithm that takes into account high dimensionality, sparse feedback and adaptive inference bias: https://arxiv.org/abs/2209.08403
A few specific products developed for optimizing the JD marketplace are described below. The main thrust of this effort is to develop ways to bring the eCommerce product and advertising markets together and unlock synergies.
Blending ads and organic content is particularly important in e-commerce as ads are embedded into content throughout the platform, implying that the impact of both ads and organic content depend on each other. This project develops a practical method to blend ads and content on e-commerce platforms that respects advertising-content interactions and balances multiple platform objectives. The method involves a deep-learning system for click-through prediction combined with a “virtual-bid” formulation for balancing objectives that is auto-tuned from historical data. The formulation is a building block for an auction payment system such as the VCG mechanism that respects externalities. Deployed on JD.com’s mobile app.
Overview: https://arxiv.org/abs/2105.13556.
Helping users discover new products is important for e-commerce platforms to serve as a healthy marketplace for brands, and also for the success of marketplace sellers on the platform. But e-commerce platforms have substantial difficulty exposing new products to their users on account of an information problem: in the early stages of launch, new products have not accumulated enough sales, orders, or other user-engagement information in order to reliably assess them of high-enough quality so as to rank them high in search listings. As part of this project, we redesign the search engine of JD to boost the rankings of new products that have been recently advertised. Our premise is that sellers’ advertising decisions reflect private information that sellers possess about product quality, and if sellers tend to advertise more the products they believe have higher quality, boosting advertised products can help surface better new products to users. A large-scale field experiment implemented on JD shows that incorporating ad propensity information into the search ranking algorithm benefits both the platform and consumers, in the short run. Our findings showcase a new channel by which advertising can improve outcomes for consumers and platforms in e-commerce, through its ability to reveal information that can be used by platforms to improve search algorithms. Also, it highlights the usefulness of economic theory-driven feature engineering and calls for blanket separations between ad and product markets to be re-thought.
Digital Couponing is particularly powerful in e-commerce. Digital coupons are complex pricing contracts that enable differential pricing to be implemented at scale in a way that is targeted, personalized, with individualized deadlines, product scope and contactual features, while reducing psychological reactance from consumers to price variation. JD’s couponing platform, which we helped refine, delivers a wide variety of coupons and vouchers to users and also helps delivers advertising.
A paper that leverages field experiments to evaluate the efficacy of JD’s consumption vouchers as a policy tool to stimulate household spending and help mitigate the economic costs of COVID-19 is https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3679362.
Overall, by leveraging causal inference, experimentation, and machine learning & AI with micro-data in a scalable way, these products help advertisers better measure and implement their digital marketing campaigns when working with publishers and to better optimize their pricing and advertising strategies, while at the same time, promoting healthy growth of the platform and its marketplace.