what are the largest argument maps created to date?

Replies to This Discussion

Permalink Reply by Jeff Conklin on April 10, 2008 at 6:52am

The QuestMap database that I am in the process of converting to Compendium for the Corporate Environmental Policy group at Southern California Edison contains over 100,000 IBIS nodes spanning 16 years of continuous issue mapping by the environmental professionals in the group. This is undoubtedly the largest IBIS corpus around, but it is far from clear what the general quality of issue mapping is, or what the size is of the average issue map. It is a goldmine of empirical data on the strengths and limits of everyday IBIS usage.

I have smaller QuestMap and Compendium projects that illustrate a variety of applications (e.g. strategic planning, system design, etc.) and degrees of integration of reference materials into the maps. Almost all of this stuff is client proprietary to some extent, but it should be possible to sanitize many of these maps so that they can be turned over for study and analysis.

Jeff

▶ Reply to This

Permalink Reply by David Price on April 11, 2008 at 3:28am

100,000 IBIS nodes spanning 16 years is a fabulous resource (and needless to say we haven't built/accumulated anything on this scale yet).

To what extent was the Corporate Environmental Policy group working to integrate and distil the nodes into a coherent (but evolving) meta-structure across the 16 years?

David

▶ Reply to This

Permalink Reply by Jeff Conklin on April 11, 2008 at 6:04am

What a great question! I don't think there was any effort to integrate or distil or make coherent. They used it as a filing system primarily, and to some extent as a place to think through an issue or create a plan, but every map was created to address an immediate need and there wasn't really time or sustained inclination to create coherence in their shared space. The root shared map is called "The Commons" and it really is a pile of trash, a classical tragedy of the commons. Nobody owned the whole space. One of the complaints about the system was that people tended to start a new map on a topic even if they knew a one already existed, because the work of making sense of a map that was created by other people for a different purpose was (and is) considerable.

The salvation of the system is Search. People go in now and do a search on "climate" or "water management" and are blown away by the rich set of maps over that period of time and the detailed story they tell. Mike Hertel, the director, loves to tell me the story of his latest case of looking like a genius to his management because in a brief study of what is in QuestMap on a topic he can form a very comprehensive and historically grounded view about what the company should do now.

I think one of the biggest challenges for the kind of coherent meta-structure that we are talking about creating on climate change is that it is very easy for the same or very similar questions to come up and be explored in slightly different contexts, and very difficult to consolidate those deliberations in a way that is robust and coherent. I'm pretty sure the cost of doing so will be prohibitive if it's all manual labor unless we can devise (i) tools that help users create that coherence quickly and easily when they are trying to accomplish a task, and (ii) power tools that help editor/moderator/librarian types keep up with hygiene and (infra)structural maintenance.

I would sure love to do more detailed examination of CEP usage. Perhaps there will be funding to do some retrospective studies at some point.

Jeff

▶ Reply to This

Permalink Reply by Adam on September 18, 2008 at 2:39pm

Digging up an old topic...

Jeff,

So what you're saying in paragraph two is that a mass of bad maps, with good search, are good enough for that group?

How does that search work? Is there any ranking system for those maps? Any way of saying "x map is better than y map in these areas", and weighting the search by metadata or staff opinion?

Adam

▶ Reply to This

Permalink Reply by Jeff Conklin on September 18, 2008 at 5:31pm

Adam,

That's a pretty good paraphrase: a mass of bad maps plus good search is good enough. Or ... a good, fast search engine can make up for a lack of a coherent navigable structure (but I guess this is what Internet search is all about). One piece I would add is that letting users create and fill "maps" is an important sensemaking enabler. The maps (created on the fly) define meaningful chunks of information (documents & emails) and IBIS argumentation/deliberation.

There isn’t any ranking or weighting in the system. Users create a search query using keywords, node type(s), author(s), and date range specifications, and then scan the results. They select the search results that seem promising and drill down into them, sometimes looking for a specific map or node, other times hoping for chance discovery of information relevant to their current task.

I doubt there’s way that weighting or ranking would be meaningful here. These users aren’t browsing for good maps or something interesting to read; they have an immediate task, and they are looking for background and historical material that will inform the work they’re doing on the task. Search success is about relevance to their task. Example: a staff member formulating the corporate response to proposed new legislation that will change the rules on the use of ocean water for cooling might stumble on a map with some relevant thinking that took place ten years ago in company-state agency negotiations over environmental mitigation of environmental impacts of a nuclear plant. My hunch is that it wouldn’t really matter to that staffer if that map had been ranked as “good” or “badly organized” or anything else. What mattered was the relevance (a complex cognitive judgment) of the material to their task, and their ability (skill plus software capability) to locate items likely to be relevant.

One more thought about maps as a unit of organization. Maps in QuestMap and Compendium are user-created collections of nodes and links. Maps are created and named to deal with a topic, like "SONGS [their nuclear plant] environmental mitigation". Many such maps then contain submaps of meaningful chunks, and so on, so maps correspond to chapters and sections and so on. When users search the massive “issue base” at Southern California Edison they are looking for relevant maps. They may find a node that looks promising, but then they follow the (transclusive) link from that node to the map (or maps) containing it to get its fuller context.

Does that help? I didn't go into how search works in Compendium, which because of the transclusion feature has some interesting elements. I'm guessing that your reason for asking is about the challenge of giving users ways to manage the scale of argumenation stuff as it becomes immense.

Jeff

▶ Reply to This

Permalink Reply by Adam on September 19, 2008 at 12:11am

I agree ranking can be difficult and that some creative solutions will be necessary, but I think it wouldn't be too difficult. For example, it is possible to rank search results based on common paths before searching. Also, one thing that doesn't exist in compendium is the idea of resource ranking. Namely, if the attached files to a node are ranked highly useful, then that node would have a greater weight among two equal alternatives. Then again, users could vote on the quality of the entire map itself, or it could be an aggregate of the item node and resource ranks within that map, ensuring that search results are more likely to come from a commonly useful map than not. There are a ton of possible combinations and algorithms for search in this case.

The big thing that I'm concerned about with search is making it -very- easy for users to see a similar search result and suggest that the result be merged into the current map, or vice versa. The difficulty of merging/linking nodes will likely correlate to the number of chaotic maps. Assuming a user base of the general public, there will be more critiquers than creators, so there will be plenty to cast votes for node positions and sort out the details of merging maps, or just chunks of maps.

Basically, the idea is that any decision we might make in organizing a map can be broken down to a series of simple votes. Those votes result in maps organized by the public, for the public, a few nodes at a time. It would become a slow iterative process of refining the maps, just like wikipedia does with its articles, but I'm willing to bet that given enough time, the average group of users will figure out how to make it intelligible.

What capability to do the users of the Edison maps have for merging/linking them? I'd be curious to know the rate at which they do. In that scenario, however, I suppose they don't want to disturb the historical record of a meeting. Whereas in a massive global system, the meeting would never really "end".

Transclusion will be interesting. Giving folks the ability to say "show me every question reverse osmosis is an answer for" should return some really fascinating results.

▶ Reply to This

Permalink Reply by Jeff Conklin on September 19, 2008 at 7:47pm

Adam says "What capability to do the users of the Edison maps have for merging/linking them? I'd be curious to know the rate at which they do. In that scenario, however, I suppose they don't want to disturb the historical record of a meeting."

The QuestMap & Compendium capability for merging/linking maps is the use of transclusions between them, or more often simply transcluding one into the other (or into each other). SCE staff do this fairly often (although not with a great deal of rigor or precision) as way to collect related items when they are pulling information together on a given task.

As for preserving historical accuracy, they do try not to work directly in maps which were created to document meetings, though sometimes questions raised in a meeting give rise to follow on exchanges right in the meeting map. (If these exchanges go on very long the after-meeting material is usually chunked into its own map.) But many maps are created by one person working on thinking thru and collecting information about a task or project, and of course any kind of merging/linking is normal in such maps.

"The big thing that I'm concerned about with search is making it -very- easy for users to see a similar search result and suggest that the result be merged into the current map, or vice versa."

This is indeed a huge potential for creating coherence in these semi-structured issue bases, and for adding value (albeit perhaps more work) to a user's mapping experience. It all comes down to the contextual relevance of the found material to the task at hand -- too many false positives and the feature is likely to be ignored or turned off.

Compendium has an option called "Auto label searching" that identifies identically worded nodes as you are typing the label, but I find it to be not very helpful for most issue mapping situations because the semantically important matches rarely have indentical strings at the beginning of node labels, i.e. lots of false positive matches. (It's also unbearably slow in a database with over 100,000 nodes!)

Jeff

▶ Reply to This

Permalink Reply by Robert Parks on November 15, 2008 at 6:53am

I don't have an answer to your question, Mark. But it occurs to me to add a couple of questions. (You can delete them if they're extraneous to your intent in questioning size/scale.) (1) How can argument maps be joined (or federated)? If we take the notion of integrated systems seriously, no matter how large a map is, won't it imply links with other maps? (2) Is it important that the maps embody an "argument", rather than a "topic" or "issue"?
Bob

▶ Reply to This