 |
All right, so, the iPad tracker had a rather… interesting… weekend, and the original post’s comments are becoming impossible to sift through (if you’ve got as-yet-unapproved comments, sorry, I’ll get them approved today). We’re reaching out to the community of people who are interested in (annoyed with?) our iPad tracker to help us make the next – God willing final – evolution of the methodology.
So, here’s what’s happened thus far with regards to the iPad tracker, complete with a timeline of our updates:
Sunday April 4th – iPad tracker launched. Our original algorithm, we used unique IP addresses across the Chitika network to calculate the estimated number of iPads in the wild. How we did it: we tracked all new unique IP addresses that showed an iPad and divided by the percentage of the Internet we see at any given time. We also divided by the number of IP addresses we expected each iPad to visit. The problem we ran into: as iPads were owned for longer periods of time, the ratio of IP addresses to iPad shifted dramatically. Result: overestimation of about 40%.
Thursday April 8th – Steve Jobs announces at the iPhone OS 4.0 press conference that about 450,000 iPads have been sold as of that day. Our tracker said about 650,000. Our first re-calibration, we stuck with the IP tracking but changed the multiplier to incorporate how many IP addresses each iPad is expected to see. We used Jobs’s announced number to get that multiplier. The problem: that multiplier is most certainly not static, and grew pretty quickly. Result: overestimation again.
Monday April 12th – Chitika engineer and iPad owner Gui discovers a workaround to the issue (to us) of Safari not allowing third-party cookies. Counter is re-calibrated again, and IP tracking – with the unpredictable growing variable of IP addresses per iPad – is swapped out for cookie tracking. You can see Gui’s methodology for this at his post on the Chitika engineering blog. Problem: the cookie-in-safari method is imperfect; Safari turns out to be much more of a pain to deal with – cookie-wise – than we originally thought. A good number of iPads register multiple cookies. Result: as of Saturday night, the numbers had become overinflated again.
Saturday April 17th – With the ultimate bad timing, just after the cookie-counting ticker rolls over the 1 million milestone, Chitika’s engineers re-calibrate the counter again, down to about 700k, based on much stricter and more conservative multipliers, also using a combination of IP addresses and cookies. People following the tracker are, rightly, very ticked off… no announcement was made. We improved the tracker, but didn’t tell anyone, and let it fall after a massive milestone.
The state of the tracker now: we’re running our “Cookie 2.0” methodology of the tracker, which takes IP addresses into account and shifts the whole tracker to a more conservative (and correct, we hope) frame of computer-mind. As of this posting, the tracker is at 804,550. For comparison’s sake (and because it’s still running locally on our servers), the original predictor based on IP address is at about 1.8 million, and the Cookie 1.0 version (the one that was running until late Saturday night) is at about 1.2 million.
Now, we’re reaching out to the community – what can we do to display the data we have in the best possible way? Every time we’ve re-calibrated, it’s been from a new development on the back-end of the tracker device that should make the counter more accurate. Admittedly, we’ve been dropping the ball on letting people know about the changes ahead of time – that’s not good, and I apologize. So what we have is the following:
- Unique IP addresses with an iPad
- Cookied iPads
- Knowledge that what we see at any given time is about 15% of the Internet
- A historical timeline of the above data going back to the iPad’s launch
How would you use that data, and how far off do you think we are from what Apple will say this week? Also, do you think our methodology for getting around the Safari cookie issue is correct, or do you know of a better way to do it?
And I am very sorry for not informing people of changes to the counter before they happened. We got obsessed with making the tracker better, and didn’t think about communicating why, how, and when that would happen. We want this tracker to be as accurate as possible. It won’t be perfect – estimators never are – but we want it to be as close as computerly possible.
Looking to add some zing to your writing? Poll the People is here to help! We offer a free signup so you can test our service and see how we can help you find the perfect tone for your content. Whether you’re writing for a blog, a website, or just for fun, we can help you find the right voice for your writing. So why wait? Sign up today and start writing with confidence!
My suggestions on presentation of the data:
(1) Bar chart with new unique iPads seen by day, with an optional link to a numerical table with the values rounded to the nearest thousand for each date.
(2) Some basic insights into change over time of those values, like new iPads on launch weekend, first week, second week, etc.
(3) Line chart (and optional numerical table) with the daily observed ratio of unique IP addresses over unique iPads.
(4) Similar daily bar chart as (1) but this time aggregating unique iPads by the date it was last seen, instead of the date first seen. This should give a sense of how many iPads have not been seen in a while and for how long, a possible indication of cookie resetting, returns, etc.
Thanks again, and hope this methodology stands a better chance.
-d
“Knowledge that what we see at any given time is about 15% of the Internet” may not help you much, as a user is bound to hit that part of the internet sooner or later. Well, that is the problem. The user who just got a new device may have a 15% chance to be visiting that part the internet at first, but a month later it is quite likely that she has already made a visit even if it is a brief one. Of course, I presume your part of the internet is fairly random, not places that only attracts people who are interested in some niche, let’ say games or stamp collectors.
I think it is far better to get a snapshot of US based browsing devices over, let’s say a weekend, calculate the ratio of iPads to all devices visible on your network during that period and then multiply it by a reliable estimate of the actual total number of US based browsing devices. Again, I assume the part of the internet you can see is a random selection of websites.
Thank you so much for finally coming out and explaining everything. That shows you care. Here’s what I’d like to see from your data:
1) If possible, can you back adjust all the data to show iPad sales per day on a single scale? I’d definitely like to see sales per day starting from launch. If you can’t back adjust because you don’t have the info, just show separate charts using each method starting from when that method is used.
2) For the per state information, could you give the option to sort by ascending people per iPad, since that’s a more meaningful metric.
Thanks guys,
– Jeff
P.S. I’ll reiterate what I said in previous emails – it’s less important to me how accurate your site is in an absolute sense than to see accurate trends in sales. That’s why I was so annoyed that you kept changing the count.
[…] reliable, and often we were seeing the same iPad with multiple cookies. I’ve put up a post detailing the changes that have been made to the tracker since its launch, when they were made, and why they were made, […]