Studio Blog

"There is always something to be found if you look for it."
--Anonymous

When I talk to writers and filmmakers who work for Demand Studios—believe it or not, I run into them all the time…in fact, I’m related to one or two—one of the most common questions I’m asked is why the categories of titles are so confusing and often inaccurate.

The simplest explanation is to just say that it isn’t easy to categorize 60,000 titles a day, so our margin of error is a little bit higher than most. But I’ll step out of the way of my pride and admit it, our assignment categorization needs improvement. 

Thankfully for those of you who are concerned enough to be reading this, help is on the way…but before I go into it, I would like to give a little explanation about the task in front of us.

The Essence of Taxonomy
In the Internet era, where entering information into search boxes has become the preferred method of locating information, you can be forgiven if you’ve taken for granted the ability to find what you want in seconds.

At Demand Studios, we need to pair up thousands of assignments with thousands of writers and do it quickly. This requires a comprehensive category taxonomy that is large enough to hold our content without being so big that it becomes a complete nuisance to writers and filmmakers looking to find work. (And btw, the word “taxonomy” has always terrified me. Just say it aloud a couple times and you’ll see what I mean.)

Every title we create, and if you’ve been reading this blog this week you know we’re talking about tens of thousands a day, then needs to be matched with the category that fits the best. Sounds simple, but let me ask, where would you categorize “How to Find a Divorce Lawyer?” In Law or Divorce?...See what I mean.

In truth, this doesn’t explain why you might find an assignment about marriage in the “Computers” category, but there are other complications at play. Namely that the assignments categorized correctly are found and claimed far more rapidly than those that aren’t, so there is an uneven ratio of miscategorized content that sticks around and piles up as the other stuff moves through the system.

A third and final problem with categorization is that in the past we’ve relied heavily on automated programs to choose categories for titles based on contextual semantic matching. What the heck is semantic contextual matching? Let’s just say matching words to other like words. And it’s obviously not perfect, which leads to lots of titles sitting in the wrong category.

Yeah, yeah, so I’m telling you what you already know—that there are flaws in categorization. But what are we doing about it?

The Future of Categorization
It’s taken a lot of time and effort to analyze our process and figure out where the loose ends are. In order to fix the problems, we first had to find out how wrong we were, which meant recategorizing a huge sample of assignments one by one using paid professionals.

From there, we overhauled our contextual matching program and built in a process by which the program can get smarter every time we find a category that is has missed. We’re putting this new program in place next week, and we expect an immediate 15% improvement on the categorization accuracy of all new assignments. Just trust us when we say this will be noticeable.

Next we’ve had to create a plan for using this new program to put a more precise categorization on the millions of titles we already have in our reserve. This is another step that we will undertake within the next 15-30 days, and again, we expect a very clear and noticeable improvement.

Finally, we’ve had to figure out a way to ensure a higher overall level of precision and accuracy in all the areas where there might be a potential loophole (i.e. the Law/Divorce example above). The best way to do this, we’ve determined, is to get more human eyeballs on our titles. We’ve relied heavily on a combo of humans and machines in other parts of the studio (we’ve even made Terminator analogies…ok, maybe that was just me). So now we’ve decided to incorporate a better human-computer blended approach to title categorization.

What does it all mean?
Thankfully, we expect an immediate, drastic improvement in the category accuracy of new assignments in the next 10 days. By end of February, we expect that same uptick to be replicated on older assignments that have been available in the tools for a while now. And by the end of March, once humans are fully integrated, we expect that less than 1 in 25 assignments in the tools will be miscategorized.

I want to personally thank all of the contributors to Demand Studios for their patience as we’ve worked our way through myriad new frontiers of online content creation—most of all those of you who’ve managed to suffer the frustration of spending your valuable time trying to find assignments in our tools, rather than on doing what you do best, executing them. The categorization issue has been a tough nut to crack, from top to bottom, and we do realize how it impacts everyone. We hope the solutions we’ve come up with will start making your jobs easier in the next week or two.

And I’d also like to add a special thanks to the titling team for all the great information posted on the blog this week. We wanted to make sure the creator community was aware that there is a whole crew of us working hard every day to make sure there are ample assignments and hence money to be made. My hope is that we shed some light on what was previously not understood. And if we didn’t, of course, you can always contact us directly at TitlingTeam@demandstudios.com.