Constructive Interference

Craig Hubley recently wrote a long comment on our initial “Brief status” post, which raised many interesting and relevant issues. I’ve now responded in a series of six posts, quoting most of Craig’s comment and adding my thoughts. Since I wrote them “forward”, you’ll have to scroll down, or end up reading them “backward”. Happily, each stands fairly well on its own, so the order you choose is not so crucial, but now you have been warned.

Filed by Jed Harris at 7:32 pm under Status & plans
1 Comment

Taking these points in reverse order, in (9) Craig refers to interesting resources about which I know little or nothing, so I’ll just quote him without comment:

Finally are sociosemantic web considerations. For that I’d suggest the book “Ambient Findability” or (much better) the UN 1993 State of the Future Report from the American Committee for the United Nations University, which covers this in great depth and deals with the various ways delphi methods, semantic webs, sociological and psychographic categories (e.g. “Islamist”, “socialist”, “feminist”) tend to determine aggregation, order, credential, etc. It’s a surprisingly operational description.

Or read this.

Craig’s point (8) raises a number of issues on which I can comment briefly:

Usage varies more by geography than anything else, and the need for discernment of a geographic kind tends to vary not by who we are, but where we are. I need much more detail about a place in California, if I’m actually in California looking for it, than if I’m just asking about it from London, England. I think there must be room here for aggregations to vary and be finer or looser grained based on how “far” we are (in space or in time) from making a decision. Trying to pretend knowledge is spaceless and timeless has a name: scholasticism. And in a word, it’s crap. Temporal and spatial database considerations need to be there from the beginnings, or this will not be useful to guide situated effective action in the real world. In other words, no one will want to read it on a BlackBerry, and that’s where most of the interesting “driving problems” (note the sysygy, that’s “driving” as per Fred Brooks’ “so hard you solve other problems by solving it”, and also as per “moving around in the real world”). If space and time tags are wrong on the first cut, you simply cannot fix them at all later - they’ll lack integrity. As a simple example of the problems, “Toronto” was a very different place in 1998 than in 1999 (the entire 416 area code merged into a new “megacity” whereas it’d only been one of five cities and one borough beforehand). If you have a tag that says “Toronto” on something about North York prior to 1998, it’s about a neighbouring city. If it’s from 1999, it’s about the same city. This matters much more if you are heading into that city than if you are considering just visiting it next year. And, if time has to be on the tagging to determine how our tagging habits/style changes, then, we’re very exposed to semantic errors, e.g. failing to put time zone on date, leaving something which can’t be resolved to an actual time span, other than stochastically.

I don’t think we have a general way to solve the problems Craig refers to, and I’m not sure we can really address them at all. However in one respect we may be able to offer help. I would tag items “Bangkok” as a potential tourist, and our system, if it works well enough, will reflect that in building my classifier. Someone else, living in Thailand, might tag items “Bangkok” because they are analyzing real estate deals in several cities. Then, in building a social landscape, my tag should be grouped with other tourists to southeast asia, whereas theirs should be grouped with other Thai real estate investors.

This is an ambitious goal, and it may well be beyond our grasp. But it indicates our approach toward problems like the one that Craig brings up. Our users are ultimately the ones who decide what distinctions are important to them. They know more about the subtle structure of the world than we can ever hope to capture in a system of categories or attributes. Our goal is to create an environment in which they can make their most important knowledge and values available to the system, and through it to other users, with as little overhead and distraction as possible. Much knowledge and values are tacit, and often could not be made explict with any amount of effort. So to minimize the burden on users, and to engage these tacit sources of knowledge and values, we have to learn from each user’s actions without requiring them to explain themselves.

The example of Bangkok illustrates this tacit component of tagging. Neither I as a tourist, nor the Thai resident as a real estate investor, would be able to explain how we use the term “Bangkok”, but our actual use reflects our interests and knowledge. If the system can learn from that, it will help us find the material we want to see, and potentially help us understand where we fit in a larger social context, and find others whose interests mesh with ours.

To the extent that distinctions of era, locale, etc. form a vital background to the users’ way of understand some topic, the system should capture that, and ultimately help the community reflect on this initially tacit aspect of their diverse perspectives.

Filed by Jed Harris at 7:21 pm under Status & plans
1 Comment

Responding to Craig’s point (7):

Mediawiki has an irreplaceable base of content, and if there is going to be a testbed, it really has to be in that format. It’s relatively easy to build front-ends to mediawiki content that enrich it (nationmaster.com, wordiq.com, wikinfo.org are all good examples, and metaweb.com was exploring doing it also). But far more important, there’s one million articles there in English alone, including cross-links to the same concept in other languages, disambiguation pages listing all meanings of a word, deep heavy linking to make it easy to tell in what sense a word is being used (simply by comparing occurrence on pages that we are following, to occurrence on the parallel Wikipedia article). This is open content (GFDL) and thus available for any such use. There’s no other comparable source for collectively understood undisputed definitions of relatedness of terms/words/phrases. A thesaurus can complement it easily, but, a thesaurus doesn’t tell you the exact translation of the name of a particular region of Europe into nineteen languages.

There are two very interesting issues here: (1) How we can support wikis and Mediawiki in particular, and (2) How we can leverage large content repositories and Wikipedia in particular. I’ll address them in that order.

Supporting Wikis: Our thoughts about tagging have mostly been about item streams, like blogs, comments in discussions, RSS feeds, etc. The need for filtering is very apparent in those cases, and that was our original motivation. In some sense the material on a Wiki (and Wikipedia in particular) is typically “pre-filtered”. However since we are helping users to tag based on their interests, and large compilations like Wikipedia are so big that users might be unable to find the pages most related to their interests, it seems that our technology could well be relevant. I’m hoping that we can find partners who want to explore this by building a tagging front-end to Wikipedia (as Craig suggests) or to other substantial compliations.

Leveraging existing compliations, taxonomies, etc.: As Craig points out, existing compilations such as Wikipedia already encode an enormous amount of “world knowledge” and expert judgement about a huge range of topics. It seems likely that we can mine such compilations to improve our services in various ways. Craig’s example of thesaurus-like support is highly relevant.

The experience of statistical document retrieval, automatic summarization, and various other kinds of statistical text processing provides some cautionary results, however. It has proven difficult to improve the performance of such systems by integrating thesauruses and other knowledge sources. Intuition is not a reliable guide in this area and experiments are always required, no matter how reasonable an approach may appear.

I see this as an important direction for exploration once we have an initial system working well enough to generate value for users and test sites. I hope that we can attract researchers with a variety of interests in statistical learning, text classification, collaborative filtering, etc. I also hope to attract a variety of open source developers. Both groups (preferably working together) can leverage our working code and any data our users are willing to share. This seems like the kind of idea that is likely to attract their interest.

Filed by Jed Harris at 5:35 pm under Status & plans
1 Comment

I don’t have any comments on Craig’s point (4) about search engines, since I don’t know how our work will affect them.

Regarding our software architecture, and how it ties into existing systems (as discussed in Craig’s points 5 and 6):

We very much agree with Craig’s statement: “Whatever architecture is chosen, it has to find the path of least resistance through [existing mechanisms] and be dead simple.” Our goal is to come up with a back end tagging library that can be shoehorned into the widest range of existing (and future) front ends with the minimum disruption. Each front end will have to provide some UI to allow users to apply tags to items, and we’ll have to have a SQL database to store our tag info. Other than that, we can have a very narrow relationship to the host environment.

We’re planning to design the specific embedding in consultation with our first few test sites. Of course to implement and test the library we’ll have to embed it in our own feed aggregation environment, but since we control all the pieces, that doesn’t raise the same architectual issues. However our initial experience with our own environment will give us a basis for collaborating on the design with our test sites.

Craig mentions the possibility of standard APIs or protocols for tag manipulation. We will probably get to these, but I’m most comfortable trying to write standard APIs once we’ve built a few diverse working implementations. Otherwise we’re trying to design based on our fantasies about how people will use this, which are likely to be wrong.

There is another domain where standarization will be even more important, but will probably also take longer. When multiple sites are using this kind of individual tagging, it will be very helpful for them to have a way to operate within a shared social landscape, let individuals extend their profile across sites, and so forth. Interaction between sites requires careful design, since it raises lots of issues about user privacy, security, control by each site of how much information it shares, etc. Also, the protocol for interaction has to be a standard, since it will be used by multiple different developers, and has to be implemented consistently to provide interoperability.

While this cross-site interaction has the potential to generate a great deal of value, we have to realistically defer this design until we have enough experience and enough different developers involved to get it right.

Filed by Jed Harris at 4:41 pm under Status & plans
1 Comment

Continuing with my response to Craig’s comment, his third point:

Because they’re political, the influence of our politics on our tags probably has to be (if not explicit) easy to determine by correlation. Someone who’s using tags like “Peak Oil” or “culture of life” at all, is clearly part of some group or movement. Mention of “seal hunt” as a specific or distinct topic suggests concern with it as an issue. Varying terms like “monetary reform”, “capital base”, “reserve rules”, “Bretton Woods”, “dollar hegemony”, all suggest different angles on the same problem, some of which (”reform”) suggest action should be taken. The people tagging may NOT all want to find each other, but the reader DOES want to find all the angles on these issues and therefore would PREFER that tags aggregate in certain ways. For someone “on the left”, perhaps “abortion” aggregates with “women’s rights”, while “on the right” it aggregates with “culture of life”. There must be respect for these choices, and there must be ways to keep these aggregations (”redirects” in wiki-speak) under the control of the user, or a user-chosen, user-trusted, agent. At the highest level of abstraction, I’d simply choose metaphors I wished to reinforce or move towards, and those I wished to abandon, and let aggregation occur as a function of those choices. At least, it might decide which of a long list of hits to drop off, or set some ordering choices. That would be no more insidious than what google is now doing, for its own reasons (not mine).

This point raises a number of technical issues about our approach. They are worth discussing, but realistically, we don’t yet know how we’re going to handle them, so any response at this point is speculative. But hey, speculation is fun!

First, we definitely plan to give each user the ability to make their own individual decisions about how to aggregate issues. On the other hand note that that does not require them to make all the decisions themselves, we certainly plan to let them use the aggregation done by others.

Second, as far as possible we’d like the system to implicitly acquire each user’s current preferences, rather than making users “explain” what they want or why they want it. The approach we’ve adopted is to let users attach tags, and then try to learn the attributes of the content that are statistically common to the way that user uses a given tag. In some sense this should let us describe the user’s current “rule” for applying that tag. The “rule”, of course, can change gradually or abruptly over time, and we should be able to track those changes.

Now let’s consider how to achieve Craig’s design goals within this framework. First, we very likely can find the collection of people who have similar “rules”. For example, if one person uses the tag “Peak oil” and another uses the tag “Energy crisis”, and they have tagged different collections of articles for some reason, but their implicit “rules” are very similar, we can recognize that. Our ability to group people together doesn’t depend on how they spell their tags, or which specific items they have tagged, but on the statistical similarity of their tag use.

On the other hand, we don’t currently have any plans to analyze the terms used in the tags themselves. So if one person used the tag “women’s choice” and another used the tag “baby killers” but they had very similar patterns in using these tags, we couldn’t detect that they had opposite feelings about the material. We would just see them as having similar “rules”. I think current computational linguistics doesn’t give us any way analyze tag names accurately enough to avoid this limitation.

Because we start with the individual user I think we can have some confidence that how things are aggregated will remain under their control. Each user determines the interpretation of their tags. If others use tags that are spelled the same, that won’t change how your tags are interpreted. (Note that this is not at all true of existing shared tagging, which may lead to some confusion.)

The harder question in our approach is how to let users group together, share tags, and influence each other’s tagging “rules”. Because we can find users with similar tagging we can help them to group together if they want. At this point it is less clear how to show users the “landscape” of other users with similar tagging patterns, or how best to give them control over their connections to other users in that landscape. I think once we have the user base, tagging data, and technology to work on those questions, the really interesting part will begin.

Filed by Jed Harris at 1:47 pm under Status & plans
1 Comment

Continuing my response to Craig’s comment, his second point was:

The influences on how we look up tags and use them are just as complex as the influences on how we use nouns or adjectives. In addition to our own “style” or “habits”, there are “masters” (like dictionaries, or exemplary opinion leaders) who we’ll prefer to copy because we wish to have powers like theirs to make distinctions and discernments “just like they make”. A good example any software engineer will understand is the use of particular APIs: no one really wants to be first to learn them, and only if they get good reviews or become indispensible to employment, do we want to take the overhead of learning them at all, let alone mastering them. Taxonomy is high investment.

So, consider that the direction in which we are desiring to modify our own habits or style, is a key factor in tagging: tag choices have at least a direction in time: there’s more distinction in this area over time, less distinction in that, adoption of standard terms here, abandonment of non-functional legacy terms there. This suggests we need at least the time to be associated with a tag’s correlation with a given user in some given task - and that we might extinct the older data over time, and allow users to express preferences to tag “more like Clay Shirky”, “less like Craig Hubley”, “less like Microsoft”, or “more like the Communist Party of China”. Yes its political - categories and taxonomies always have been, always will be.

Craig’s point about “masters” is an interesting one and brings up a central issue related to the discussion of taxonomies vs. folksonomies. His example of dictionaries illustrates it nicely. Written language existed for thousands of years before we had dictionaries. Presumably writers learned how to use words in a literary way mainly by imitating others, especially those regarded as particularly exemplary. Dictionaries were only possible because there was an existing consensus on the meaning of words, along with a substantial historical trail showing the evolution of usage. So this consensus was historically prior to explicit definitions.

APIs (software definitions of values and operations that others can use, for those who are not developers), and before them mathematical and some scientific terminology, take the opposite path. Someone defines the meaning of a term (such as “set” in mathematics, or “force” in physics) and others have to learn the definition as given, ultimately creating a consensus if the term — and the theory it helps to express — is useful enough. Terms evolve to some extent, but basically the definition preceeds the consensus. But note that these terms get defined within a highly organized community that already operates under a broader consensus on what is important, and how to define and solve problems. Of course this consensus evolves, but at each step most of it survives.

The implication of these examples is that if we have a community with a strong consensus on how to do things, then putting forward new terms and frameworks by definition can be efficient (though this requires high investment by adopters, as Craig notes). However while a community is forming and negotiating a consensus, commitments have to be soft and largely implicit, and creating new frameworks by definition is not likely to work very well. Right now, our focus is on these early phases of consensus formation, where many potential participants have different, more or less incongruent ways of organizing the material, but where they share enough common concerns so that they want to work together. We aim to create systems that help these participants move toward an effective consensus with the least possible stress and friction.

One interesting question this raises is how we can detect when a community consensus is solid enough to start introducing explicit frameworks. However we don’t really need to answer this — people are going to keep trying to define frameworks, and when the consensus is solid enough, these attempts will gain traction.

As discussed in my commentary on Craig’s first point, I agree that we need to let users’ previous actions diminish in influence over the present as they fade into history. I also agree that we need to let each user view tags as influenced by the choices of other designated users. Craig’s additional suggestion that users could choose for this influence to be negative (”less like Microsoft”) is probably a good one. All of this needs empirical feedback, and the discussion helps us to figure out what possiblities to try, and what feedback to look for.

Filed by Jed Harris at 1:05 pm under Status & plans
1 Comment

Craig Hubley wrote a long comment on our initial “Brief status” post, and I’m going to write several posts responding to his specific points. His first point was:

Any means to “learn individual tagging style” implies use of past tagging decisions, and respect for those - which suggests that the algorithm’s recommendations (similar to Amazon’s, I’d presume) constantly pull the user towards their past style, and could inhibit them learning new styles or even adopting new tags. Treating tagging patterns as a deliberate choice, or “style”, suggests too strongly that they are informed decisions as opposed to evolving habits. So I’d change this language and “track individual tagging habits” rather than “learn” a “style” - avoid implying that something’s worth learning, or was chosen deliberately, or has anything to do with user individuality. I think tagging is new enough and hard enough to master, that any user who wants to make good use of new tools, will be learning new habits, discarding old ones, and will rarely want simply to continue their current “style” and have software “learn” it. I suspect it’s far more likely that they’ll want to pick one or more “masters” doing exemplary tagging and prefer to follow the tag styles they recommend.

This raises a number of intertwined issues.

First, Craig is absolutely correct that each user’s tagging will change over time. In addition, the stream of content the user is tagging will change over time. We have to be careful not to nail down any decisions based on a user’s prior actions. (I’m avoiding the word “choices” here, since we don’t really know how much the user made a “choice”.) This is a crucial difference between rules and statistical learning — rules require prior thought and tend to be hard to change. Classifiers created by statistical learning are softer and track current actions, so as the user and/or the content changes, the classifier will change.

Second, many of the questions Craig is raising are ones we want to explore empirically. How stable a user’s preferences are, how happy users are with the classifier judgements, etc. are very much questions we want to test. Within broad limits we can tune the learning to match user preferences (how fast it forgets, how sharp the thresholds are, etc.)

Finally, Craig’s point about taking advantage of other users’ exemplary tagging is potentially very interesting. We have to get basic tagging working, but our next step after that is to let users benefit from tagging decisions by others. There are many ways to do this. The easiest (for us) is to let users see the content from anothers’ point of view — presumably the sort of “master” Craig mentions. We can do this as soon as tagging is working. A more complex option is to figure out what users are “near” each other based on similar tagging patterns. This could automatically let new users get the benefit of other users more experienced tagging.

Again, all these more advanced options need empirical testing and user feedback. I’m sure in the process we’ll learn a lot about both what is technically feasible (and cost effective) and, more important, how users experience the system and what they find appealing.

Filed by Jed Harris at 12:21 pm under Status & plans
1 Comment