Bias in Marketplace Listing Enhancements
A few days ago I made a rather strongly worded post on the SL Forums regarding the Bias present in the way Listing Enhancements are displayed to visitors. I have spent a fair amount of time watching and mentally noting the ads seen, but I’d never gone the extra step of actually logging and analyzing the data. Well, I’ve been raised under the credo that strong words must be backed up by facts. So today’s post is the result of that fact gathering.
Getting The Data
The first step I took was to write a PHP Script that would pull the Marketplace Home Page and log the Listing Enhancements presented. I chose to do the retrieval using a logged out session so that I could get basically unbiased data. My reasoning was that if a user was logged in, they would be presented with ads that might be slanted toward their shopping habits or past history. (Although I seriously doubt the Marketplace has that much sophistication in its coding, but that’s another matter.)
Each time the Home Page is refreshed or retrieved, a new set of 20 Listing Enhancements is chosen and loaded into the page. A “Scroller” is used to display those images. That’s a web device that shows a subset of images then scrolls on a timed basis. In the case of the Listing Enhancement Scroller, the first three images are shown immediately then at specific intervals they are scrolled to the left and the next three images are displayed. Keep in mind that being placed into the Scroller constitutes a ‘View’ in the Marketplace vernacular. In reality only the first three images are seen right away. In practice, only the first set of three is ever seen; very few people bother to actually scroll through the full set of 20.
I then sat and pulled 1000 page loads of the Marketplace Home Page. That gave me a database of 20,000 Listing Enhancements and their display rates. From there I could work some numbers to find out how often a specific Listing Enhancement is shown.
High-Speed Intro to Randomness
At the heart of any fair and unbiased display routine is a little computer gizmo called a “Random Number Generator” or RNG. They’re used all over the place, and most advanced programming languages offer a system function that generates a pretty well randomized sequence of numbers. Once you have a random number, you can use that to index into a list of things and pull out one … basically at random. That’s how the display routine for the Marketplace should work when choosing which Listing Enhancements to place into the Scroller on the Home Page.
However it’s also possible to Bias or “Weight” the results by different methods. Instead of using a source list of Listing Enhancements where every ad is placed into the list once, you could create a source list where specific ads are added more than once. That would guarantee that specific ads get higher exposure than others. Of course that’s just one method, but suffice to say that any alteration of the randomness by which images are chosen would apply an unfair bias.
High-Speed Intro to Random Distributions
Any time you analyze a random sequence, you get what’s called a “Random Distribution”. When you have a small set of possibilities, you should get essentially an even number of selections in the results. For example, when flipping a coin, since there are only two possible results, the counts for each should wind up nearly equal as you accumulate a larger test set. However when you have a lot of possibilities, the distributions will assume a shape typically called a “Bell Curve”. It’s given that name because its shape resembles a bell. In the case of Listing Enhancements on the Marketplace, unless we had a result set in the millions, the shape of the results should take on the typical Bell Curve shape. And that’s almost what we see .. almost.
Here is a portion of the graph I created using the statistics gathered:
(click to enlarge the image – the full size is 969 x 434 pixels)
The Full Data
You will of course notice that the above is just a segment of the full graph. Wanna see the full thing?
(click to enlarge – the full size is 1676 x 858 pixels)
Along the left-hand edge of the graph can be seen what are called “Outliers”. These are data points that don’t follow the expected Bell Curve random distribution. Outliers are expected to some extent with random number generators, but the more and more samples you have, the fewer that should be present. This is why I accumulated the data over 1000 page loads; I feel that’s a sufficiently high number of samples to even out any anomalies in the RNG. The fact that we have a very well-formed Bell Curve for a majority of the data is proof of that. But still there are those danged Outliers.
I should also point out that the listings that are shown as Outliers pulled way ahead early on in my testing, and remained consistently high all throughout the test runs. To me that indicates a definite bias in the results.
Buying Lots of Ads
One of the conventional bits of wisdom regarding Listing Enhancements is that the more ads you buy, the better your chances of receiving top exposure. While this would seem to make sense, it is also is a sign of bias in the way Ads are shown. To my way of thinking, the small Merchant that can only afford one Listing Enhancement should have exactly the same chance of their ad being seen as someone that buys 100 or more. And this is what the data expresses too; the one Merchant that purchases over 200 different Ads gets their ads shown in an exactly random fashion. So far so good.
But the ones with the highest display rate are those that buy only one Ad. In fact, the display rate for a small handful of Merchants is 3-4 times the average rate. To me, that indicates a high degree of bias in the way Ads are chosen for display.
The Chosen Few
No, I’m not going to name names here. That’s not my purpose. Instead I will show you a graph that illustrates just how biased the display rate is:
(click to enlarge the graph – the full size is 1678 x 858 pixels)
As you can see, one product in particular received a whopping 11 views across the 1000 page loads. It’s also interesting to note that the Merchant has only that single item advertised. They actually have a much larger store and selection of items, but they only purchase the one Listing Enhancement.
The next level of exposure, 10 Views each, are owned by only 5 Merchants, again each of whom purchases only one Ad each. Again these Products are only one of a much larger selection available from each Merchant, but they purchase only the one Ad each.
Conclusions
I intend to spend some time over the coming weeks running tests of 1000 and possibly more page loads just to verify my initial results. But from my initial testing, it is clear to me that specific Merchants are receiving a very favorable exposure rate. It’s also interesting to me that each of them purchases only the one Ad even though their selection of available products is much more extensive.
I should also point out that the size of one’s store is not a deciding factor in the display rate. Many Merchants with much larger inventories are given much less favorable exposure. And the one Merchant that pays for over 200 Ads gets only average exposure. So the factors that one would expect to influence display rate … don’t.
I leave it to you to determine for yourself what factors are being used to bias the display rate. For me? I’ve made up my mind. And my conclusion doesn’t shed a very positive light on Linden Lab and the Marketplace Management Team.
Comments
6 Comments on Bias in Marketplace Listing Enhancements
-
Medhue on
Sun, 23rd Dec 2012 10:56 AM
-
Nita Bracken on
Sun, 23rd Dec 2012 1:38 PM
-
Darrius Gothly on
Sun, 23rd Dec 2012 1:43 PM
-
Darrius Gothly on
Sun, 23rd Dec 2012 1:45 PM
-
Magnet Homewood on
Mon, 24th Dec 2012 10:50 AM
-
Czari on
Sat, 29th Dec 2012 10:51 PM
I wonder if the time of day matters. As I’ve said in the forums, I consistently see higher sales in the early part of the day, and then almost no sale or few sales in the later part of the day. Being that sales on the MP make up the majority of sales now, I would have to assume that something strange happens to the MP after a specific time of the day.
Did you only test the homepage enhancements? I tend to stick with landing page since I get better exposure there. And in turn better sales.
@Nita – Yes, just the Homepage Listing Enhancements for this first pass. I am planning on testing a few other Category Landing pages though. Do you have any suggestions as to pages you would like to see tested? Personally I’m leaning toward the higher-traffic (and presumably higher utilized) categories such as apparel and accessories, but I’m open to specific suggestions too.
@Medhue- That’s a good question; I don’t know. I did my initial testing in the wee hours of the morning because I didn’t want to gank bandwidth from real customers. I guess my next test sweep will be for a Category Landing page … and sometime around midday.
Interesting results. I have realized that listing enhancements don’t do jack all for me to get my products seen, my sales are about the same for the products that had an E than without, after a few weeks. Sales used to be better, but now I go through regular cycles of a week of good sales, and a couple of weeks of poor sales, and I can’t be bothered (and don’t have the know-how) to research why that may be.
If you could bottle that script up (with instructions) to be able to send by email, I’d be willing to refresh the marketplace page 1000 time at a set time in the day for my timezone (European) and send you the results, if any of that is possible.
Since I began back in 2007, even before I began selling anything, there were allusions to “favored merchants” on the forums…and this was in the old Resident Answers, not even a forum dedicated to merchants or merchandising. This was also referencing in world stores since XStreet wasn’t utilized quite as much as the current MP.
I’m not surprised by these results, but it certainly does add to the overall annoyance I have with LL in general at the moment.
DGP4SL Staff
-
