Jon Udell

Information technologists often recite David Wheeler’s famous aphorism:

Any problem in computer science can be solved with another layer of indirection.

Often, though, they omit the corollary:

But that usually will create another problem.

Those problems used to plague only IT folk. But now we’re all involved. Effective social information management is quite severely constrained by the fact that regular folks are not (yet) taught the basics of computational thinking.

For example, when I explain my community calendar project to prospective contributors, they invariably assume that I’m asking them to enter their data into my database. It’s quite hard to convey: that the site isn’t a database of events, only a coordinator of event feeds; that I’m only asking them to create feeds and give me pointers to their feeds; that this arrangement empowers them to control their information and materialize it in contexts other than the one I’m creating.

I’m having some success explaining this model, but it’s slow going. People don’t take naturally to the indirection and abstraction.

Here’s another example. I know various folks who are trying to create online resource directories of one kind or another. I’ve identified a pattern, which I call collaborative list curation, that is an ideal way to solve this problem. Consider this directory of blogs for the Monadnock region. It looks like any other such directory, but it’s made differently. Again, there is no explicit database. Entries come from the del.icio.us tag delicious.com/judell/monadnockblog — a personal collection whose items are, currently, the same as those in the global collection delicious.com/tag/monadnockblog.

I’m subscribed to the global collection at feeds.delicious.com/v2/rss/tag/monadnockblog which means I can monitor it for new items, vet them, and transfer those I want to include to my personal collection. If I wanted to delegate that editorial control, I would point my directory-making service at the del.icio.us account of a trusted associate and have it camp on that account’s monadnockblog tag instead of (or in addition to) my own.

Of course this is all way too indirect for any normal person to grok, which is why nothing has been added to the global collection. Even many IT-savvy folks, I’m finding, don’t take naturally to this model.

That said, I’m finding that once I can get people to walk through one of these experiences, and see the connection — OK, I do this over here, and that happens over there, and it can also happen somewhere else, and I’m in control — the light bulb does go on.

Now we need to take forward-thinking evangelists like me out of the loop, and get people to discover for themselves how to wire the web. If Live Clipboard didn’t exist, we’d have to invent it. Oh wait. It doesn’t, and we do.

This week’s ITConversations show suffered a tragic glitch that rendered the audio unusable, but I was able to transcribe it as text. My guest is John Leeke, a carpenter who takes care of old buildings and shares his knowledge of the tools and best practices involved in doing that. His methods of sharing have evolved over many years. He started in the early 1980s as a writer for magazines like Old House Journal and Fine Woodworking, transitioned to Internet publishing when that became possible, and more recently has become a leader in the use of Internet video to communicate knowledge that’s embodied, as he likes to say, in the mind, the hands, and the heart.

His approach to Internet video exemplifies and weaves together a number of themes that I’ve focused on in recent years, including narration of work, online apprenticeship, tacit knowledge, screencasting to document our work in the virtual world, and video to document our work in the physical world.

JU: We got introduced by way of the folks at the Open University, whom I met when I visited the UK in January 2007 to speak at the Technology, Knowledge, and Society conference. They were showing me their FlashMeeting videoconferencing system, and they cited you as an example of somebody who’s making very practical use of the medium in your work, which is historic home renovation.

JL: Right. I’d been using FlashMeeting for about a year and half then. They had singled me out because I wasn’t doing education, or developing the FlashMeeting system, like they were there at the Knowledge Media Institute, I was out in the real world doing things with it, demonstrating the horizontal movement of knowledge.

JU: That absolutely grabbed me. Ever since I got involved in Internet video, I saw there was a huge opportunity for horizontal, or direct, or peer-to-peer transfer of knowledge. In particular, of knowledge that is embodied, literally — it’s in your hands…

JL: It’s in your mind, your hands, and your heart. I’ve been sharing what I know through print media since the early 1980s. I grew up working in my father’s shop, in the 1950s, and then was out in the field working on historic buildings as a preservation carpenter for fifteen years. Then I fell into writing about my work: Homebuilding Magazine, Fine Woodworking, Old House Journal. I got pretty practiced at that by the late 1990s.

JU: You’ve published books too, right?

JL: Yes, I’ve self-published a series on caring for older buildings. Through the 1990s I knew that video would be important for my work, but I never came around to publishing anything in video. I didn’t have the time or dollars to put into it. But but 2003 and 2004, it was getting streamlined enough and easy enough to do over the Internet.

JU: As much use of online video as there is, I think we’ve barely scratched the surface when it comes to the sort of sharing of practical knowledge that you’ve been doing.

JL: It’s starting to happen. Just yesterday a colleague sent me a link to a YouTube video about how to draw and sketch the classical forms, like Ionic capitals. It was an architect showing how he sketched, and how he developed a balustrade for a fancy classical building. It showed him actually doing it. This wasn’t happening in the 1990s. You could do it, but it was a huge expensive production. Now you can do it for a couple of hundred dollars, and sometimes even less.

JU: Of course there’s still the question of why someone would do this. And in fact, the theme of the talk I gave at that conference was network-enabled apprenticeship. The idea was that throughout human history, people have learned trades and crafts by direct observation and imitation.

JL: Yeah, workers working side by side. And it’s more than observation. It’s the guiding of hands that makes that work. Internet video, even when it’s live, doesn’t get you all the way there. But it’s certainly a dramatic next level beyond print media, that’s for sure.

Expositional work online — presentation of words and pictures and even videos — it’s all presentational. Someone develops it, and as a separate event in time someone else comes and watches and learns. But when it’s live and interactive, that’s when you jump to the next level. Being there in person is best, of course, but this is a really valuable and powerful intermediate level because it opens up access to many more people than I can get together with personally, side by side.

JU: Can you give an example?

JL: In our work we’re often restoring old windows. This is the time of year when you have to take care of them. One of the details of that work is reglazing, where the glass meets the sash — the wooden frame that slides up and down. There’s a material called glazing compound, or putty, and it’s easy enough to use so that any handy person can do it, but it’s hard to get it so that it looks nice and smooth and even, if you haven’t done it before. Once you learn, it’s a cinch. And it’s easy to show someone how. I’ve taught eight-year-olds and eighty-year-olds how to run a perfect line. But you can’t do that with even a detailed series of photos.

JU: And you’ve tried…

JL: Yes, I’ve written three or four articles over the years, and each one is better, and you can learn a certain kind of thing from print and photos. You can learn what kind of putty to use, you can learn how to hold the putty knife. But until you see a putty knife in motion, and can respond in realtime — adjust the angle, a little more pressure — you can get it in thirty seconds if you’re side by side, and in a few minutes over interactive video.

JU: So you’re talking about a couple of levels here. The first is direct observation and imitation. My first revelation on that front was when I had to fix an old HP laser printer. I found a parts kit online that came with a video on a CD, and it enabled my to successfully disassemble and reassemble that printer. Later I realized there was no other way I could have done the job successfully. No written instruction would have gotten me there.

JL: Right, that’s one level and it works well when the printer you’re repairing is just like the one in the video. And when the job involves mechanical parts that lock and fit together.

But with the window putty, it’s different. You’re working with a plastic material. It’s as if you had to make those printer parts yourself. It’s basic stuff, not manufactured stuff.

JU: The motor skills are subtler, and the nonverbal communication is more critical.

JL: Right, and with the nonverbal communication as well as the visual, you really need to be able to go back and forth between the learner and the teacher. If you can do that within seconds — or if you’re standing next to someone, microseconds — that feedback between the eyes, the mind, the hand, the muscles, the tool, the material the tool is shaping — that’s how they learn so fast in person. And it can happen in seconds when you’re doing interactive video over the Internet.

JU: What’s your setup for doing these interactive training sessions over the Internet?

JL: I take my notebook computer, plug in my Sony HandyCam, and shoot whatever it is we’re teaching or discussing. It’s getting to the point where it’s all plug and play, and if I can do it, many people can.

JU: So that’s the broadcast piece of it, what’s the setup for interacting with people who are following along?

JL: That happens on a page at my website, HistoricHomeworks.com. Other people log in there to the FlashMeeting system, and if they have camera and audio at their end, I can see and hear them. Typical numbers are two or three participants, up to eight or ten. The live sessions are also catalogued for later viewing.

JU: The FlashMeeting system has some interesting features, including a method of visualizing the conversation so you can see who spoke when and for how long.

JL: That helps support the Knowledge Media Institute’s principal mission, which is to study and understand how knowledge spreads from person to person around the world. The analytical features built into FlashMeeting serve that mission.

It fascinates me. For example, you can see displayed on a map of the world the locations of viewers of these recorded sessions showing how to restore historic windows, or painting and restoring exterior woodwork. I can see where the interest is, and it turns out that people everywhere care about this stuff, because there are wooden buildings all around the world. On six of the seven continents there are people using these videos streaming from my office in Portland, Maine. At KMI they joke that they’re waiting for someone to start watching in Antarctica.

JU: It’s an interesting point because in the world of online media there’s a lot of emphasis on what’s new, but you’re operating out on the long tail. Your piece on interior storm windows was very relevant to me because I just went through the exercise of doing the stretch-and-seal method, and your demonstration of how to build reusable interior storms really got my attention.

That’s an idea a person might never encounter. But if you do, it doesn’t matter when. The publishing world calls this evergreen content, it’s valuable anytime.

JL: Right. There’s also a discussion on my website about this topic. It’s more expositional — words and pictures — and that goes hand in hand with the video. One of the limitations of the FlashMeeting system is that I can’t annotate the video, after the fact, with links to those materials.

JU: A lot of folks will look at this and say, OK, John Leeke is an unusual guy. He doesn’t just do the work, he also documents the work, and that’s great for him, but it’s not really relevant to most people who won’t have the time or inclination. For them, this process seems tangential.

But I think that’s often untrue. Here’s an example. I have a pellet stove, and there are a couple of maintenance procedures that I frankly screwed up the first time through because I didn’t absorb the understanding of how to do them from the manual. What struck me was that once I knew how to do it, I could have illustrated these procedures with a couple of five minute videos. And maybe I should just do that myself. But the thing is, if I’m the dealer, and I’m getting complaints from customers who are buying these things and then failing to understand the manual and screwing things up, it’s very much in my interest to do some of my own video documentation.

JL: Of course. And by the way, I’m not special. I’m just a carpenter up here in Maine, taking care of my own house. It just turns out that my work is also helping other people to take care of their houses. Well, yes, it’s not unique but special that I have this compulsion to share what I’m learning and figuring out. But the ability to share it — well, no matter who you are, if your neighbor sees you fixing your windows, and comes over and knocks on your door and asks about how to do it, you would show him. This is just an extension of that. Now we can have neighbors further afield.

JU: Yes. There was a time when the work people did was visible. You saw what they did.

JL: You saw what the people next to you did.

JU: That’s right. And you understood what the different kinds of work were, because you saw people doing that work. But then, in the industrial age, dad went off to work, he disappeared in the morning, and showed up again at the end of the day, and work was a black box. Who knew what dad did?

JL: That’s the industrial disconnect. And there’s a disconnect on the marketing side as well. Through the last half of the 20th century, as the industrial revolution gears up to grind itself into nothing — which is now happening — the method of marketing to more people than needed stuff was to disconnect the people from each other, so that everybody needed something, instead of sharing with their family or neighbors. Everybody needed their own lawnmower. But you figure your lawnmower is sitting idle in your garage for 99% of its time. One lawnmower could easily mow everybody’s lawn on the block.

But that’s the consumer culture that was developed by manufacturers. So very few people now know to run that glazing compound to seal the glass to the wooden frame. This is purposeful. They don’t want people to know how to run glazing because that limits the market for vinyl plastic imitation windows.

So I only have one person on the block I can teach locally, but I can connect with more people with interactive video. Because of the access to the long tail, I can be teaching lots of people who need to know that.

JU: Here’s another aspect I wanted to ask you about. When it’s hard to see how work is done, it’s hard to know what it’s like to be a person who does that kind of work. Unless it’s in the family, you won’t see it, and even then you probably won’t. You don’t have the family or community scope in which to see other kinds of work being done. And lacking that, you can get pretty far down an educational path before you realize that the path isn’t for you at all.

JL: Right. So, I’ve been focused on task-specific demonstration, but you’re talking about another thing that’s happening with video over the Internet — life blogging, or life broadcasting. I don’t think anybody’s doing that as a tradesperson. What is it like to wake up at 4:30 AM, so you can be on the site working on the windows, all day long, and then get in your pickup truck and drive back home? As you say, a lot of people could go all the way through school, and study building construction at the college level, and then take specialty courses in historic carpentry work, and by the time they’re in their early 20s they’re well-educated and have a good set of hands-on skills — and then realize that they don’t like to get up early in the morning.

JU: You’ve painted the downside, and that’s fair, people should understand that, but on the upside, the life blogging should also communicate how you feel when you drive by a house that you’ve restored, and how you know the people living there feel as a result of the work you’ve done.

JL: Absolutely. This is the heart side of the work that the industrial revolution leaves out. It boils everything down to mind and hand, and leaves out the heart. That is the heart side, when you drive by those buildings you helped restore, last month or last year or 20 years ago. It is the reason why we get up early in the morning to go to work. You know that you’re helping people who live in and use those buildings.

JU: Now there are certainly many people who will feel that these methods they get paid to practice are proprietary knowledge they wouldn’t want to reveal. My argument is that in a lot of cases, by demonstrating expertise you’ll attract more work than you lose, and that it’ll often be more interesting and rewarding work. What’s your experience?

JL: Both of those ideas do play strongly in the building trades. It’s a real tradition to keep secrets. Going back hundreds and hundreds of years, with the guild systems, there were ways to control the sharing of that kind of knowledge. And it’s still the case. Not every plasterer who can do those decorative Ionic capitals wants everybody to know exactly how they do it. But they do want everybody to know that it can be done.

You’re right, this is how artisans can do good marketing — by letting people know what is involved, by showing some of these methods, and they don’t have to give up all their secrets in order to do that. But you can help people to understand that it’s not just a machine spitting out product, it’s people making stuff with their minds and their hands and their hearts.

That’s another part of how I use Internet video. I go to some of my colleagues’ shops, as well as my own, and show what this is all about, because it is not well understood by the public. Video can get to the nuances of the heart side of this work.

JU: Also, if you can show me how to take care of some basic things for myself, maybe I can turn around and hire you to do something really special.

JL: Yeah. I’m hoping that we’re now in a post-modern cultural movement, which is what I think you’re talking about. Back in the 1970s I was already working in this realm of making fine things by hand, and there was a groundswell of interest. That’s when Alex Haley’s Roots phenomenon happened. It was important because it touched the hearts of people in America. That’s really what our restoration work is about, it’s the connnection with the people who once lived in these buildings. It wasn’t the national trust and the President telling us to save buildings, it was people who wanted to save them because their grandfathers built them.

JU: So where do you fall along the continuum of trade secrets and knowledge sharing?

JL: I’m at the extreme end of sharing everything I know. I’m a one-person microbusiness and always have been. I grew up in the midwest where sharing what you knew, and helping people, was what life was about, for everybody. That was the culture. It was a natural for me. It didn’t seem like it was worth keeping secrets.

My dad said that if you want to do well in trades, you have to let people know what you do. This is what it’s been all about for me — letting enough people know.

JU: And you have found incredible marketing power in doing what you do?

JL: Oh yeah. As I was working as a tradesperson in the 70s, and a contractor in the 80s, I made a shift because I’d been doing a good job of documenting my work. That’s something else I learned from my father. I also had the documents he created for his work, going back to the 20s, this huge information resource that I had to share.

JU: Really? What did he document?

JL: He documented his work in the arts and trades. He was a commercial artist through the 20s, then shifted into furniture and buildings at the craftsman/artisan level.

JU: And he left behind detailed logs of his practice?

JL: Yeah, detailed files of every project he ever worked on. So I learned that as part of my carpentry and woodworking, growing up in his shop, and continued it when I left his shop and came east to work on old buildings. So by the early 1980s I had this whole backlog of my own work to share. And by sharing it, I created extraordinary interest in my work. Back then it was through the print media — Fine Woodworking, Old House Journal, Fine Homebuilding — and a lot of people learned about the work I was doing to restore columns on old porches, saving windows, doing woodwork repairs. When I learned something I thought was worth sharing, I’d write an article about it. The editors loved it, and their readers did too, it was the authentic stuff, what was really going on out there in the field.

With that body of knowledge, by the late 1980s I was consulting on projects, helping people solve problems with their buildings. That meant I could be on even more projects, helping more people, and if I was writing about what I was learning, then each project was an order of magnitude larger. If I’m doing hands-on work on buildings I might only be helping a few people. If I’m consulting, it might be tens of people. If I’m writing, we figured ten or fifteen thousand people were using my articles. Each is a jump in magnitude. Then of course the Internet, where I got an early start in 1994 and 95.

JU: I’m sure a lot of folks will look at your example and feel that, since they’ll never become featured writers for magazines, there’s no point in doing this kind of sharing in a more modest way. But I think there’s benefit at any level of engagement. You’ve clearly thought through the dynamics of the communication pattern here: one-to-many, multi-level distribution. But for a lot of people, even with electronic media, that isn’t obvious. They’ll still spend a lot of time doing one-to-one communication. They’ll write something up, they’ll even take some pictures, but then they’ll just email that to somebody else.

JL: Two birds with one stone. I realized that if I wanted to accomplish the things I want to get done in my life, I have to get more than one result for every action or activity. The print — and now online — publications that I do are my marketing program, so I don’t have to spend money on advertising. And now you call, and want to talk with me, and if I was only getting one benefit from that, I wouldn’t be able to say yes. But I can already see two or three things that’ll come from talking to you, so I can say yes.

Say I’m thinking of taking on a project to help my neighbor rebuild her front steps. OK, I can earn some money. And I can take a series of photos for a print article, and that’ll bring some more income but it’ll also help with my personal goal of sharing more, and then I can easily shoot a little video that I can broadcast on the Internet and that will help an astonishing number of people on the Internet. So I can’t say no, because I’m getting multiple benefits. But I would have to say no if the only benefit was getting paid to fix the steps.

JU: You’ve really thought it through.

JL: The key is that the video camera and the computer and the Internet are just tools, no different from my table saw and push stick, or my old wooden hand plane. They’re all just tools, and they’re all in the same kit for me, and I’m a tool user, and I help people with their old buildings.

How can people do this? I’ve found a balance. Instead of watching television, I make television.

JU: Well said. Thanks John!

As I mentioned here, I’m exploring the viability of Python as a way of programming the newly-announced Microsoft cloud platform, Azure. Partly that’s because I love Python, but mainly it’s because I believe that the culture surrounding Python and other open source dynamic languages can fruitfully cross-pollinate with the culture that infuses Microsoft’s platforms.

One of the reasons these cultures face each other across a great divide is religious attachment to low-level operating systems. In the cloud, though, the differences among these low-level systems are increasingly hidden behind interfaces to higher-order constructs: compute nodes, storage objects. These, in turn, are building blocks for still-higher-order services that will be created — and consumed — both by platform vendors and by the developers who are their customers.

It becomes possible, in this new world, for platforms to support a continuum of access styles. You want object-oriented? Do it that way. RESTful? Go for it. You know the Python or Ruby libraries best? Use them. The .NET Framework? Use that. Or even mix and match according to convenience and taste.

Consider this Python module written by Sriram Krishnan, which wraps the RESTful interface to Azure blobs. It’s written in standard Python, using OpenSSL-based cryptography. When I tried it on my machine, though, I ran into an inconsistency in my local Python installation.

Normally a Python developer would debug and fix the installation. But I was planning to deploy this module in IronPython on Azure, and IronPython doesn’t run compiled modules such as OpenSSL. It can, of course, use equivalent .NET functionality — in this case, the method implementing the SHA-256 flavor of keyed-Hash Message Authentication Code. So I made that small change.

At this point, having eliminated my module’s only dependency on unmanaged code, I thought I could run it in the Azure development fabric, and then deploy it to the Azure cloud. But no. Azure’s security model currently won’t allow Python even to import pure-Python modules at runtime. A wacky solution might be to use Python’s custom import mechanism to load those modules over the network. More practically, the modules might be provisioned into Azure.

I don’t know how this will play out. Meanwhile, there’s another option: Eliminate all use of Python modules, and rely only on the .NET Framework. So as an experiment, I switched over from Python’s minidom, httplib, time, and base64 modules to their .NET equivalents.

The good news is that this works. I can deploy the module to Azure, and use it in the cloud. The bad news is that, in some cases, I’d rather use the standard Python modules. The .NET equivalent to Python’s httplib, for example, is the HttpWebRequest/HttpWebResponse pair. But these APIs differ from those provided by httplib in a couple of ways that annoy me.

First, there’s an inconsistency in the way headers are handled. You get and set most headers using the Headers collection. But you get and set a few special ones, like Content-Type and Content-Length, using special named properties.

Second, status codes are handled inconsistently. Most responses return status codes. But for codes in the 4xx series, an exception is thrown.

To me these behaviors are quirks that make it trickier to create RESTful interfaces. I’m sure there are reasons for them, and people who prefer them for those reasons, but I’d rather just use httplib. In any case, if both styles are available, there’s no need to argue. Everybody gets what they need.

We’re not there yet in the current Azure preview. Those of us chomping at the bit to run IronPython in the cloud will have to be inventive. I expect things will get easier as both Azure and IronPython mature, and as Python technologies like Django and NWSGI are — I hope — woven into the fabric.

Why might this matter? Again, I’m looking for cross-pollination. Python culture will be able to make really productive use of higher-order Azure services such as identity, access control, workflow, Live Services. And it will also exert a positive influence on the future evolution of the Azure platform.

It was my great privilege to interview Carl Hewitt for this week’s Innovators show. He is principally known for work dating back to the late 1960s and early 1970s, when he helped lay the foundations for a declarative, message-oriented model of computation. Then, and for decades thereafter, the virtues of that model were not widely appreciated because the problems it solves were not evident. Now, in an era of multi-core systems, cloud-based computation, and global interconnectivity, it makes all kinds of sense.

In this conversation, we review the themes Carl sounded in this recent talk at Stanford. (Video is here, and an audio-only version I made for myself is here.)

In one of the most striking moments in that talk, Carl says:

What can I change? Just me. For anything else, I send a message, I say please, and I hope for the best.

Then he laughs and adds:

Does this sound like some circumstances you are familiar with?

Having thought deeply, for 40 years, about the intersection of computation and human affairs, he has arrived at an elegant synthesis: The same organizational and communication patterns govern both realms. As well they should, since the two are now and forever intertwingled.

At the end of our conversation, we turn to Carl’s critique of Wikipedia. He raises important questions about how Wikipedia’s cadre of mostly-anonymous administrators, dedicated to the codification of conventional knowledge, come into conflict with academics and researchers whose work pushes the boundaries and conflates the categories of conventional knowledge.

In response to an item last week about regional sources of imported oil, @jesperfj wrote:

Not sure what to conclude? Do informed people like Udell really not know that?

I really didn’t. And the reaction to the item, plus my survey of friends and associates, tells me that while some informed people did, many did not.

From this, I know exactly what to conclude. Like all complex systems, our civilization is buggy. We need many eyes to make the bugs shallow, and there all kinds of things that the brains behind those eyes can’t know a priori. But with the right kinds of mental prosthetics, we can learn rapidly and bootstrap ourselves into a position to reason effectively.

Data visualization is a crucially important mental prosthetic. But we’ve yet to evolve it much beyond the graphical equivalent of the wooden leg.

Consider this chart:

It’s a somewhat useful way to visualize the fact — counter-intuitive for many — that the Middle East ranks only third among suppliers of oil to the U.S. But here is a much more useful way to visualize the fact — intuitive for everyone — that the Middle East is where most of the world’s oil reserves exist:

What do you call this kind of projection, where country size is proportional to a variable? It’s the sort of wickedly effective graphical device that we should all want to be able to deploy, at a moment’s notice and with minimal effort, in order to make sense of data and reason about the world.

Like Tim Bray, I’m angry about “the financial professionals who paid themselves millions for driving the economy into a brick wall at high speed, then walked away while we pick up the pieces.” But I’m also angry at myself for visualizing, way too late, along with the rest of us, the magnitude of the giant pool of money and its constituent flows.

We could have seen more, seen better, and seen sooner. In many domains, as we go forward, we will have to.

Last week on Interviews with Innovators I spoke with Nova Spivack about Twine, a service that’s been variously described as the first mainstream semantic web application and “just del.icio.us 2.0″. You’ll find support for both points of view in my conversation with Nova. It’s true that, unlike del.icio.us and other comparable services, Twine is built squarely on top of what Nova calls a “semantic web stack.” But it’s hard to discern, in Twine’s current incarnation, just what that entails.

One of the bookmarks I imported into Twine, for example, is http://www.educause.edu/HEBCA/623. It’s the home page for an organization called the Higher Education Bridge Certification Authority (HEBCA). In Twine, the item shows up tagged as an Organization. That’s the kind of thing that you’d expect a semantically-aware service to do. But what does it mean for Twine to classify HEBCA as an Organization? It’s unclear. Here’s the offered link. It points to a small collection of items that mention HEBCA, but Twine does not “know” anything at all about HEBCA.

What our conversation revealed, though, is that my method of testing Twine — which involved importing all my del.icio.us bookmarks — was flawed. I had assumed, incorrectly, that Twine would absorb the bookmarked pages themselves. It will, but it doesn’t yet, currently it only aborbs the del.icio.us metadata such as title and link. If you want to find out what Twine’s linguistic and semantic analysis can do, you need to pump content into the system.

That’s easier said than done. The only API available for content injection is email. Twine materializes a private email address to which you can send items you want to post as private notes.

I spent a few minutes thinking about writing a script to automate the injection of items I bookmark on del.icio.us. It’s doable, of course, but only by dint of hackery that I would undertake grudgingly and normal folks would never imagine or attempt.

This kind of integration will get a whole lot easier for everyone when the various services export events representing our actions within them. For example, a couple of weeks ago I reorganized my del.icio.us tagspace, adding the tag socialinformationmanagement to a group of otherwise-tagged items in order to emphasize that particular facet. And I tweeted:

Imagining new kind of FriendFeed event: “Jon Udell updated 9 del.icio.us bookmarks, adding the tag socialinformationmanagement”

In other words, when I perform a public action in some service — like bookmarking an item in del.icio.us, or even just retagging an existing item — the service posts an event on a topic to which other interested services subscribe. In this case, FriendFeed is the interested service. When I configure FriendFeed to monitor my del.icio.us account, it asks del.icio.us for the list of event types that it exports, and I choose which of those to display in FriendFeed.

Of course FriendFeed needn’t be only an event subscriber. It can and should be a publisher too. Another service should be able to ask FriendFeed for the list of event types it aggregates for me — bookmarking an item on del.icio.us, posting a photo on Flickr, adding a book to my LibraryThing library — and then subscribe to all or just some of those events.

(While we’re at it, I want a service that can not only subscribe to my aggregated event feed, but also take actions. One of the actions I’d configure would be: When Jon bookmarks a new item on del.icio.us, fetch the item and inject it into Twine using the specified secret email address.)

Of course there’s nothing new here with respect to basic change notification. Weblogs.com has been doing that in the blog realm for many years. Now it’s time to generalize the mechanism across the range of services that manage various aspects of our online lives.

At a party the other night, a friend mentioned that the country supplying us with the most oil is Canada. Maybe so, I said, but on a regional basis the Middle East dominates, right? He wasn’t sure, but didn’t think so. And it turns out he was right, at least according to the US Dept. of Energy data he sent me. That data says that the Middle East ranks third among our regional sources, behind North America and Africa.

Here’s the world overview for 2007 in thousands of barrels:

And here’s the regional breakdown:

The links go to regional views where you can hover to reveal per-country numbers.

When I do these kinds of exercises, I’m always struck by two things. First, it amazes me how much of what we think we know is wrong. I was sure that the Middle East was the dominant regional source.

Second, I’m always a bit discouraged by how geeky you still have to be — even with the great online tools we have now — in order to pull answers to simple questions out of raw data. When my friend cited these numbers, the first thing I wanted to know was: How do they break down by region?

I wound up using Dabble DB because I happened to know that it includes all the necessary ingredients:

• Can import tabular data from web pages
• Can drop and rename columns in an imported table
• Given a column with locations — countries, states, zipcodes — can map the corresponding columns
• Can publish views for anybody to see

This was a huge leg up! But a lot of folks wouldn’t know about that tool. And even if they did, many wouldn’t overcome some of the remaining obstacles. For example:

• Importing. There are a few different ways to grab data from a web page. You can have Dabble DB parse the page, or you can copy/paste. In this case, I wound up trying both and had better luck with the latter. But we’re still very much in an era when data published to the web is not really intended to be used as data. That first step can be a doozy.
• Sharing. After pasting in the data and reducing the table to two columns — country names and 2007 1000s of barrels — I had my answer. And if you were an authorized user of the application, I could have shared it with you. But in order to publish to the world, I had to produce a special URL. And then I realized a single one wouldn’t suffice. The shareable views aren’t interactive. You can’t drill down from the world overview to the Middle East segment. So I wound up having to create views for each region, generate an URL for each view, and keeping track of all that was confusing even for me.

Still, I’m excited. We’re really close to the point where non-specialists will be able to find data online, ask questions of it, produce answers that bear on public policy issues, and share those answers online for review and discussion. A few more turns of the crank, and we’ll be there. And not a moment too soon.

In July 1995 I wrote a column in BYTE with the same title as this blog post. It began:

One day this spring, an HTTP request popped out the back of my old Swan 386/25, rattled through our LAN, jumped across an X.25 link to BIX, negotiated its way through three major carriers and a dozen hosts, and made a final hop over a PPP link to its rendezvous with BYTE’s newborn Web server, an Alpha AXP 150 located just 2 feet from the Swan.

Thus began the project on which this column will report monthly. Its mission: To engage BYTE in direct electronic communication with the world, retool our content for digital deployment, and showcase emerging products, technologies, and ideas vital to these tasks. We don’t have all the answers yet — far from it. But we’re starting to learn how a company can provide and use Internet services in a safe, effective, maintainable, and profitable way.

Today I felt that same kind of excitement when I clicked on this URL:

http://elmcity.cloudapp.net

There isn’t much to see. But what happens behind the scenes is quite interesting to me. The URL hits a deployment in the Azure cloud where I’m hosting an IronPython runtime. Then it invokes that runtime on a file that contains this little Python program:

hello = "Hello world"

Finally, it gets back an object representing the result of that program, extracts the value of the hello variable, and pops it into the textbox.

This is the proof of concept I’ve been looking for. Now I can begin an experiment I’ve been wanting to do for a long time. I have an ongoing personal project, elmcity.info, about which I’ve written from time to time. It’s hosted at http://bluehost.com, it’s written in Python using Django, and it’s invoked by way of FastCGI.

Back in the BYTE era, I loved learning about the web by building out a live project, and explaining what I learned step by step. Now I want to explore, and document, what it’s like to build out another live project in the Azure cloud.

Could I do it in Amazon’s cloud? Sure. In fact I already did, as an experiment. And if it were cheaper to run there than on Bluehost, I’d currently be hosting elmcity.info on EC2 instead.

Could I do it in Google’s cloud? Not sure. I didn’t score an account there and can’t yet try. The interactive pieces of my application should slide nicely into AppEngine’s Django framework. But much of the work is done in long-running processes which I believe AppEngine doesn’t yet support.

In any case, it’s obvious why I’ll be focusing on Azure. I suspect, though, that my focus will be different than most. I’m not a hotshot .NET developer, just an average guy who can get some useful things done in environments that enable me to create small, simple, understandable programs, and do so in agile and dynamic ways. I think that Azure — admittedly nascent in its current form, as Ray Ozzie said at the PDC — can be such an environment. Let’s find out.

On election night, the most useful information display I found was the New York Times’ interactive election map. It’s another bravura performance from a team of talented designers and programmers who keep raising the bar. Back in May, two of them — Gabriel Dance and Shan Carter — joined me for a conversation about how they do this work, and why it matters.

Last week, the venture capitalist Tim Oren wrote an essay entitled The Newspaper Crash of 2009… And How You Can Help in which he argues:

The industry has abdicated its social function to support a well-informed electorate, and become a propaganda arm of the left. In so doing, they have sullied their brands and lost the trust of their readers. The economic consequences of this default of their value proposition are now becoming apparent. The Internet and an economic crisis together would be bad enough, but the industry has only itself to blame for the egregious behavior on display for the last few years, and at its worst right now.

And concludes:

When the lights go out at the New York Times, our work will be finished.

The newspaper industry has surely earned this kind of scathing criticism. And it may well fail to capitalize on the amazing opportunities for self-reinvention afforded by the Internet. But the Times is attracting an all-star team of information architects, interactive graphics designers, programmers, and media producers. And according to Gabriel Dance and Shan Carter, these folks are increasingly collaborating with reporters to marshall complex information in ways that make the newspaper’s stories deeper and more open to independent analysis and interpretation.

So I’ll say it differently: When the lights go on at the New York Times, our work can start.

Next Thursday is World Usability Day, a distributed event that will happen in lots of places. One of them is Putney, Vermont, not far from my home, where I’ll be speaking at the New England venue, Landmark College.

The program says:

A description of Jon’s talk is forthcoming, but we’ve asked him to help the audience further their thinking about the potential of video on the web in support of teaching and learning, as well as the the importance of the structure behind the information with which we all work, exemplified by his work on compiling disparate web resources, as in his work on Keene-related events culled from the internet and viewable at elmcity.info/events.

Great suggestions! Video and structured data are very different domains. That creates a nice opportunity to talk about key underlying principles, and relate them to the practices of teaching and learning. So, here’s the blurb.

This is going to be fun!

This week’s Interviews with Innovators explores the Granicus solution for civic webcasting with CEO Tom Spengler. If you’re lucky enough to live in a city that is a Granicus client you’re already familiar with how it works. If not, take a look at the Newport Beach, CA site. It’s a beautiful thing. You can see the video and minutes in a synchronized view, jump to the agenda items you care about, and view associated staff reports in context.

For citizens the benefit is clear. If you have access to these proceedings on cable TV — even random access with a DVR — it’s still a challenge to pinpoint a segment you care about. What’s more, there’s no way to form a URL that refers to that segment so you can share it, and so that online discussion about the segment can aggregate around that URL. Granicus gets it right. Agenda items define the natural set of RESTful resources for these meetings, and this system enables people to cite, bookmark, and link to those resources.

Behind the scenes the system enables the town clerk to annotate a copy of the minutes with timecodes, so that the data required for segmentation and synchronization is captured in realtime and available immediately upon conclusion of the meeting. That’s exactly the kind of pragmatic approach that will help make transparent democracy as ordinary and routine as it ought to be.

When friends and family ask about the Professional Developers Conference I attended this week, I tell them it’s kind of like Microsoft’s State of the Union address. I’ve been to a number of these over the years. This was my first as an employee, and Microsoft’s first as a company fully committed to what I believe are the right principles, patterns, and practices. That’s a big statement, and as always you should consider the source and take it for what it’s worth. But if you’ve followed my work over the years, you’ll spot many familiar themes in the following exegesis of the day two keynote by Don Box and Chris Anderson, and you’ll know why this PDC put a huge smile on my face.

In case you’re unfamiliar with the theatrical genre I call PDC performance art, I should briefly explain. Traditionally, at this show attended by thousands of software developers, a few of Microsoft’s technical leaders come to the stage, write small programs on the fly, and run them. These daring high-wire acts are humorous and entertaining, but also deeply informative. The live code exercises new platform technologies, and tells stories about why and how the audience might want to apply those technologies.

The story that Don and Chris told began with a simple web service, running on a demo machine, that printed out a list of processes — effectively, a Unix ps (process status) command. It was built using several key components and features of the .NET Framework: LINQ (Language Integrated Query) to query for and enumerate the list, WCF (Windows Communication Foundation) to package the query as an HTTP-accessible service, UriTemplate to control the namespace of that service, SyndicationFeed to format the response as an Atom feed, and ServiceHost to run the service on the local machine.

When it ran, this program enabled a browser running on the local machine to surf to a service running on the local machine and view its process list as an Atom feed. This colocation of web client and web service on the local machine is a key pattern that I first explored a decade ago. Dave Winer named the pattern Fractional Horsepower HTTP Server and put it to excellent use in his pioneering blog tool Radio UserLand. The pattern embodies a key underlying principle: symmetry. We have long been conditioned to think of the Internet in terms of clients versus servers (and now services), but that’s an artificial distinction. In the terminology of TCP/IP networking, there are no servers and clients, there are only hosts — that is, peer nodes communicating directly with one another. Firewalls and NATs abolished that symmetry. The newly-announced Azure Services Platform is a technology that can help us restore it.

The next step was to extend the program, adding the ability to kill any of the running processes. The Atom feed was already modeling the process list as a set of URI-addressable resources. To implement the feature in a simple, standard, and discoverable way, it was only necessary to apply the HTTP DELETE verb to those resources. Internally, the program of course had to implement a DeleteProcess method. But that method name need not, and according to RESTful best practices should not, appear in the service’s API. And happily, the service did not — as do so many purportedly RESTful services — expose any URIs that look like this:

http://localhost/service?method=delete&process=123

Instead it only exposed URIs that look like this:

http://localhost/service/Process?id=123

An HTTP GET method, invoked on this URI, could return information about the process. An HTTP DELETE method invoked on the same URI accomplishes the kill function, and does so without violating the RESTful principle of interface uniformity. Later on we’ll see a nice example of the benefits of that uniformity. But here, let’s notice another key principle at work. I’ve said that the kill operation was discoverable. That’s true thanks to the Atom Publishing Protocol. It defines a hyperlink within each entry that is the RESTful endpoint for update and delete requests targeted at that entry. So the program’s DeleteProcess method queried the Atom feed for those hyperlinks, and used their addresses to create the URI namespace that exposed process deletion to clients.

The general principal at work here is linking. A core tenet of RESTful style is that link-rich hypermedia documents, useful to people because they make it possible to navigate and discover related things, are equally useful to programs for the same reason.

These are, of course, best practices for an ecosystem sustained by web standards like URI, HTTP, and XML. But it was wonderful to see those best practices clearly demonstrated in a PDC keynote. It has not always been so. Trust me, I would have noticed.

On the next turn of the crank, the standalone process viewer and killer was network-enabled thanks to Azure technology that I first told you about a year ago, back when it was known as the Internet Service Bus. Using it, Don and Chris created this endpoint in the cloud:

http://servicebus.windows.net/services/DonAndChrisPDC

You can go ahead and click that URL if you like, it’s still live. What you’ll fetch is an empty Atom feed. During the keynote, though, Don and Chris wired that endpoint to the program running on the demo machine onstage. This was accomplished in a purely declarative way, by adding a binding to the program’s configuration file that pointed to the chunk of web namespace whose root is servicebus.windows.net/services/DonAndChrisPDC.

This wasn’t yet a cloud-based service, that came later. At this stage it was still a local service that was advertised in the cloud and made available to the public Internet. To accomplish that, Azure has to enable clients out on the Net to traverse intervening firewalls and NATs and contact the local service. It does so in a way that illustrates another key principle: policy-driven intermediation.

The need for such intermediation was soon apparent when the local service was relaunched with its Azure binding. Now anyone in the world could visit the above URL in a browser, view processes, and even try to delete one. Within seconds, someone did try, and Don shouted: “Stop the service, Chris!” There was no real risk — the program was running in debug mode, with a breakpoint set on DeleteProcess — but it was a great theatrical moment.

Now in fact, the service was secure by default. In order to expose it to the Net in an unauthenticated way, there was a configuration setting that overrode the default security. After removing that, an interactive (i.e., browser-based) request produced a login page. Crucially, that login page did not come from the local service, but rather from Azure which was handling security, as well as connectivity, for the service. The policy in effect was username/password, so after typing in appropriate credentials, interactive access was restored, but now in a controlled way. A different policy — for example, one requiring X.509 certificates or SAML tokens — could be defined in, and enforced by, the Azure fabric.

Next, the local client program that had been accessing the service — first directly, then by way of the Azure cloud — was adapted for the same kind of secure access. To do that, it requested an authentication token from Azure’s access control system, and then inserted that token into the HTTP headers of subsequent requests to the service.

So that was act one. Here was Don’s segue into act two: “Chris, are there other services in the world we might want to program in a similar fashion?”

Why yes, Chris said, and launched Live Desktop. There, courtesy of Live Mesh, were some folders that were synchronized cloud replicas of folders on the local demo machine. Since Live Mesh is also based on Atom feeds, it should be easy to convert a RESTful service that enumerates and deletes OS processes into a RESTful service that enumerates and deletes Live Mesh folders.

It was easy. In the client program, the base URI changed from servicebus.windows.net to user-ctp.windows.net/V0.1/Mesh/MeshObjects. And the authentication token had to change too because, well, to be honest, Azure’s subsystems aren’t yet seamlessly integrated. But that was it. The same LINQ query to find entries in a feed worked exactly as before. Only now it listed folders in the cloud rather than processes on the local machine. That’s the beauty of a uniform HTTP interface in the RESTful style.

Note that the Live Mesh API works symmetrically with respect to the cloud and the local client. The same program that lists folders in the cloud can list folders on your local machine. You just point the URIs at localhost, and use the Fractional Horsepower HTTP Server that’s part of the locally-installed Live Mesh software.

Note also that you don’t have to use any Microsoft technologies to work with these Azure services. The demo program used LINQ, WCF, and — for the Live Mesh stuff — a wrapper library that packages the API for use by .NET software. But any technology for shredding XML and communicating over HTTP will work just fine.

In act three, the focus shifted to Azure’s storage service. Using all the same patterns and principles, the program morphed into one that could upload DLL files into Azure’s blob store, use Azure tables to associate human-readable metadata with the DLLs, and issue a simple relational query against the set of uploaded files.

Finally, in act four, the service that had been running locally, on the demo machine, was adapted — with some minor changes — to work with the local development version of the Azure compute cloud, and then deployed to the staging and production areas of the real cloud.

To sum up, the emerging Microsoft platform not only spans a continuum of programmable devices and services, it also spans a continuum of access styles that are all based on core standards including URI, XML, and HTTP. I think this is a great story, and I’m exceedingly happy to finally be able to tell it.

I hope James Governor, Mary Branscombe, and Kim Cameron will triangulate on this, but here’s my report on a cosmically funny incident at a party last night. I walked up to James just as he witnessed Kim being forcibly denied access to the venue. He lacked the necessary identity token — a plastic wristband — and couldn’t talk his way in.

If you don’t know who Kim is, what’s cosmically funny here is that he’s the architect for Microsoft’s identity system and one of the planet’s leading authorities on identity tokens and access control.

We stood around for a while, laughing and wondering if Kim would reappear or just call it a night. Then he emerged from the elevator, wearing a wristband which — wait for it — belonged to John Fontana.

Kim hacked his way into the party with a forged credential! You can’t make this stuff up!

I use the Amazon API to check wishlists programmatically, and back in March I mentioned that it was being upgraded in a way that would break the Python wrapper I’d been using for years. Readers pointed me to a new wrapper called PyAWS, but I found that it didn’t offer the one thing I needed: A simple way to retrieve all the ISBNs on a wishlist.

I solved the problem for myself with a few lines of code, but neglected to include them. Today, that March entry received a hilarious comment:

I came here searching for a way to retrieve my Amazon wishlist using PyAWS… You’re the top query (out of a grand total of 5!) for pyaws wishlist amazon.

However, reading the blog article above, I had flashbacks of Fermat’s Last Theorem:

“After poring over this mysterious PyAWS, I found a wonderfully simple way of retrieving a wishlist like with PyAmazon. However the margin of this blog post is too narrow to contain the few lines of Python code required.”

:-)

Could you please post the said few lines of codes to retrieve a wishlist with PyAWS? Would be much appreciated. I’d rather not have to pore over the whole Amazon API documentation to learn how to retrieve a simple wishlist or two with PyAWS.

Sorry about that! Here’s what I’m currently doing. It’s not PyAWS, just a regex hack of the raw XML output from REST queries.

import urllib2,re

def getAmazonWishlist(aws_access_id,wishlist_id):
  url = 'http://webservices.amazon.com/onca/xml?Service=\
    AWSECommerceService&AWSAccessKeyId=%s&ListId=%s\
    &ListType=WishList&Operation=ListLookup' %\
    (aws_access_id, wishlist_id)
  s = urllib2.urlopen(url).read()
  pages = re.findall('<TotalPages>(.+?)</TotalPages>',s)[0]
  for page in range(int(pages)):
    url = 'http://webservices.amazon.com/onca/xml?Service=\
      AWSECommerceService&AWSAccessKeyId=%s&ListId=%s\
      &ProductPage=%s&ListType=WishList&Operation=ListLookup\
      &ResponseGroup=ListFull' %\
      (aws_access_id, wishlist_id, page+1)
    s += urllib2.urlopen(url).read()
  return re.findall('<ASIN>(.+?)</ASIN>.+?<Title>(.+?)<',s)

(Ironically the margin of this blog post is too narrow for the few lines of Python code required, so I’ve split those lines where indicated.)

By the way, DoubleSearch reveals that although Google currently finds only 5 results for pyaws wishlist amazon, Live Search finds 9. More importantly, if the blog entries from Rich Burridge and me are indeed the most relevant results, Live Search puts them first.

That’s not always true, of course. Often Google does better. But not always. In any case, even when the first pages of results from both engines are equally relevant, they’ll likely differ in ways that DoubleSearch invites you notice.

If you’re inclined to dismiss what I’m about to say because I’m employed as a Microsoft evangelist, then fair enough, move along, there’s nothing to see here. But if you’ve followed me over the years and continue to trust my instincts, then hear me out on this. I’ve always believed in, and acted on, the principle of diversity. If you think the same way, then you use more than one operating system, more than one programming language, more than one application in many categories.

So why would you use only one search engine?

If you haven’t tried Live Search in a while, you’ll find that it’s improved quite a bit. I’m not saying it’s better than Google, but I am saying it’s usefully different. Given the central importance of search, I argue that it’s in everyone’s interest to exploit that diversity.

Now arguably most people don’t care about diversity. There’s a strong impulse to find one way to do something, and then stick with it. People don’t readily adopt new behavior. To help them along, you need to minimize disruption.

To that end, I’ve been asking some friends and associates to give DoubleSearch a try. Specifically, I’m asking them to make it their browser’s default search provider, then let me know how long they keep it and, if they drop it, why.

I know there are logistical issues with DoubleSearch. In particular, given the side-by-side-in-frames presentation, it’s awkward to click through on a search result. You’d rather right-click and open in a new tab. Some people already have that habit, others don’t, their experiences will differ accordingly.

I’m sure there are deeper cognitive issues as well. For example, I find it useful to compare the two result pages side-by-side, but others — maybe many others — will just find that distracting.

Anyway, if you do try this experiment for yourself, feel free to comment here on how it goes.

Five years ago I was exploring the idea of embedding active chunks of structured data into web pages. Back then I used the phrase interactive microcontent. Nowadays, we say microformats. If you’re a reader of this blog you’re probably technically-oriented, and you already know about microformats. But most people aren’t, and they don’t. The challenge has always been to provide an end-to-end experience that will enable non-technical folks to create and use these nuggets of semantic web goodness.

Here’s a project that can help: Oomph. It’s the first lab component of the relaunched MIX Online site, which is run by Microsoft evangelists who, like me, care about web standards and web innovation.

To demonstrate Oomph, I’ve injected a microformatted event here:

I created this event using Live Writer — a WYSIWYG blog editor — and its Event Plug-in. In this case, no data entry was required because the plug-in enabled me to search Eventful and capture the existing Pumpkin Festival record found there. That’s just the sort of grease we’ll need in order to overcome data friction.

Still, for most folks there’s no obvious reason to publish a microformatted event. The information looks nice, but it’s not clear what you or anyone else can do with it.

One aspect of the Oomph toolkit is an Internet Explorer extension that makes that embedded event come alive. Here’s what this page looks like in IE with the Oomph extension:

The arrow points to an indicator that “gleams” when a page contains microformatted elements.

Clicking on the indicator opens a panel that activates them. In this case, the event is enhanced with icons for a variety of calendar import methods.

When an item has a location, you can map it:

If Oomph were only an IE-specific extension, I’d wouldn’t be writing about it. But in fact, it’s a cross-browser solution based on jQuery. I can’t demonstrate that here because WordPress.com blocks JavaScript, but consider these two pages:

1. Oomph: with explicit JavaScript. This page explicitly calls the Oomph JavaScript code, and works cross-browser. Try it!

2. Oomph: without explicit JavaScript: This page (like the blog entry you’re reading) does not uses the Oomph JavaScript code. The enhanced behavior is still available in IE, by way of the Oomph extension. It could also be available in Firefox, Safari, or Chrome if similarly extended.

It’s really helpful to have the option to go both ways: Server-side where it’s permitted, client-side where it isn’t.

There’s more to Oomph: CSS styles for microformats, and a Live Writer plug-in for inserting hCard (contact) elements into blog postings. You can get the toolkit and documentation on CodePlex. Nice work guys!

From a 2004 entry entitled Information Routing:

To further my own self-interest in keeping track of things, I’ve made a minor extension to the del.icio.us bookmarklet, so that selected text on the target page is used for the (optional) extended description of the routed item. This makes the items I route easier for me to scan. And for you too. Of course if you did the same, the items you route would be easier for you to scan. And for me too.

We keep losing sight of these basic principles. Meanwhile, their importance keeps growing. Here’s a case in point from this morning’s flow: an item in FriendFeed from Charlene Li:

OK, I get the gist. If malware and/or Facebook are central topics for me, and I haven’t heard about this already, I’ll click through. But in most cases, I’m scanning my flow to expand my peripheral awareness. I don’t have time to click through and look at everything. I need context wrapped around the items that appear in my flow.

Annotating your social bookmarks is a great way to provide me with that context. In del.icio.us, here’s how you do that:

Before you invoke the del.icio.us posting form, select the text that best summarizes the item you’re bookmarking. Then paste it into the Notes field.

Here’s the del.icio.us view of the item that Charlene and I have bookmarked:

It’s been bookmarked five times. There are two annotations. But I claim that only one of them is useful. carlhaggerty’s blurb is site boilerplate that comes from an F-Secure meta tag. It says nothing about this particular item. My blurb, however, adds useful context. It identifies the item as a Flash-related phishing exploit.

Admittedly, you have to go out of your way to expose this del.icio.us view of the F-Secure item as bookmarked by five del.icio.us users. But in an environment where syndication and information routing are pervasive, our actions have consequences elsewhere. Here’s how that same item appears to my FriendFeed subscribers:

Now my subscribers can absorb, at a glance, the additional context about phishing and Flash. Their peripheral awareness expands. Their time spent scanning their flows is more productive. And their subconscious anxiety — about not clicking through to read the majority of items they can’t possibly have time to read — is alleviated.

I would like to enjoy these benefits too, but I need your help. Please consider annotating the items you share. If you’re so inclined, here’s a bookmarklet that will help:

post to del.icio.us

(Update: As per Carl’s comment below, that won’t work because WordPress defangs the JavaScript. Another way: Create a bookmark on your links bar, edit its properties, and paste the following into its URL or Location field:

javascript:(function(){var%20notes=window.getSelection ? window.getSelection().toString() : (document.selection?document.selection.createRange().text : ""); notes=encodeURIComponent(notes); f=’http://delicious.com/save?url=’ +encodeURIComponent(window.location.href) +’&title=’+encodeURIComponent(document.title) +’&notes=’+notes+’&v=5&’; a=function() {if(!window.open(f+’noui=1&jump=doclose’, ‘deliciousuiv5′,’location=yes,links=no,scrollbars=no, toolbar=no,width=550,height=550′)) location.href=f+’jump=yes’}; if(/Firefox/.test(navigator.userAgent)) {setTimeout(a,0)}else{a()}})()

I’ve also updated it to incorporate a better way to capture the selection, per Alf’s comment below, thanks Alf!)

(Further update: Crap. How the hell do you defeat the “smart” single quotes at WordPress.com? The above won’t work either, but I’ve put draggable and copyable versions at http://jonudell.net/delicious-bookmarklet.html).

This is a version of the standard del.icio.us bookmarklet. It updates the extension I made way back in 2004. If you replace the standard del.icio.us bookmarklet with this version, you’ll still need to highlight the salient text on the page you’re bookmarking. But you won’t need to paste it into the form. It’ll pop into the Notes field automatically. Do this, and I’ll love you forever.

PS: del.icio.us: Why not make this version the standard bookmarklet, and explain why? As your bookmarks increasingly find their way into our lifestreams and workstreams, useful annotations will matter more and more.

The fun I’ve been having with DoubleSearch has reminded me how easy it is to create new search providers that plug into your browser’s dropdown list of search engines. Here’s an interesting one: FaceSearch. As the name suggests, it finds pictures of faces.

This is nothing more than a Live Search for images, with face recognition turned on. You can get the same effect by doing a regular search, and then using the Refine By options to show only color photos of faces. But those restrictions don’t persist across searches. And if you have to tweak them every time, it’s awkward to explore face space.

With FaceSearch, you can fly through sequences of face-oriented searches. For example:

1. people: yourself, friends, family, and associates, celebrities
2. emotions: happy, angry
3. expressions: smile, frown, wink
4. places: your town, New Zealand, Mumbai, Reykavik
5. ethnic groups: Maori, Pashtun
6. decorations: glasses, hat, piercing
7. hairstyles: combover, mullet, afro

It’s interesting to compare these results to Flickr searches that include the tag face. When the facial aspect of a photo is important enough to tag, you’ll do much better with Flickr. For example, it delivers wonderful results for angry face and Mumbai face. But it finds nothing for Reykavik face, whereas Live Search finds thousands of Icelandic faces.

Update #1: It helps if you spell Reykjavik correctly. When you do, Flickr finds 300 Icelandic faces.

Update #2: As per Bill’s comment below, Google has a syntax for face search too. Hadn’t known that. So this can become another kind of DoubleSearch. Done! Now twice the fun!

Back in 2005 I made a screencast that showed how the convergence of GPS and online mapping enables us to collectively annotate the planet. The Tracks4Africa folks have been doing that since 2000. On this week’s Innovators show, Johann Groenewald explains how some GPS enthusiasts who are passionate about exploring, documenting, and preserving Africa’s rural and remote “eco-destinations” have created an annotated map that travelers can use and enhance. The GPS maps have evolved into a commercial product. The annotations — including photos and commentary — are available at the Padkos website, and also as a layer in Google Earth.

I found out about T4A when a reader commented on an earlier item about ground truthing and crowdsourced mapping. T4A is a wonderful demonstration of that possibility. It’s also a great story about how open data contributed by a community, and commercial data managed by a business, can thrive in a symbiotic relationship.

Paul Pival noticed a problem with the browser widget I made the other day to search Google and Live side-by-side. The service invoked by that widget, at dualsearch.atsites.net, fails when your query contains double-quoted phrases.

It’s an easy fix as I’ll demonstrate here. There are three ingredients:

• A itty-bitty web application
• A simple XML file
• An even simpler HTML/JavaScript file

Let’s examine them.

1. The web application just receives a query, URL-encodes it, and interpolates it into the template for a web page that invokes the two search engines in side-by-side frames.

There are a million ways to do that. Here’s a Python/Django implementation:

def doublesearch(request):
  import urllib
  q = request.GET['q']
  q = urllib.quote(q)
  template = """<html>
<frameset cols="*,*" frameborder="no">
  <frame src="http://www.google.com/search?q=__QUERY__" />
  <frame src="http://search.msn.com/results.aspx?q=__QUERY__" />
</frameset>
</html>"""
  html = template.replace('__QUERY__',q)
  return HttpResponse(html)

2. The XML file contains an OpenSearch description that invokes that little web application, passing it the query that you type into your browser’s search box. Here’s an example that uses a sample service I’ve located at my elmcity.info site:

<?xml version="1.0" encoding="UTF-8" ?>
<OpenSearchDescription xmlns="http://a9.com/-/spec/opensearch/1.1/">
<ShortName>DoubleSearch</ShortName>
<Description>DoubleSearch provider</Description>
<Image height="16" width="16" type="image/x-icon">
http://jonudell.net/img/doublesearch.ico"</Image>
<InputEncoding>UTF-8</InputEncoding>
<Url type="text/html"
  template="http://elmcity.info/doublesearch/?q={searchTerms}" />
</OpenSearchDescription>

3. Finally, here’s the HTML file that encapsulates the snippet of JavaScript that installs the OpenSearch widget into your browser:

<a href="javascript:window.external.AddSearchProvider
 ('http://jonudell.net/doublesearch.xml')">Add</a> the
 DoubleSearch provider.

You can Add the DoubleSearch widget and try it for yourself. Unlike other variants I’ve found, this one doesn’t wrap any cruft around the side-by-side results. It simply presents them.

As I mentioned the other day, I’m finding that combining the top 10 results from both engines makes for a more useful set of 20 results than taking the top 20 from either.

With today’s wider screens, placing the two result frames side-by-side works out pretty well. In this mode, however, you’ll want to avoid clicking through directly on a result. Instead, right-click on the result and open it in a new tab.

In a recent series of items I discussed ways of turning an Internet data feed into a video crawl for use on a local public access cable television channel. In the last installment the solution had evolved into an IronPython script that fetches the data, writes XAML code to animate the crawl, and runs that XAML as a fullscreen WPF (Windows Presentation Foundation) application.

This week we finally got a chance to try out the live feed, and we didn’t like what we saw. For starters, the animation was jerky. The PC that became available for this project is an older box running Windows XP. I installed .NET Framework 3.0 on the box, and it now supports WPF apps, but not with the graphics acceleration needed for smooth scrolling.

Even with the smooth scrolling that we see on my laptop, though, it wasn’t quite right. This application displays a long list of events, and it’s going to grow even longer. We decided that a paginated display would be better, so I went back to the drawing board.

We’re happy with the result. It displays pages like so:

                  Community Calendar

 06:30 PM open/lap swim  (ymca) 

 07:00 PM Caregiving for Individuals with
   Dementia (unh coop extension) 

 07:00 PM Vicky Cristina Barcelona (colonial
   theatre) 

 07:30 PM Faculty Recital-Jazz (eventful: Redfern
  Arts Center) 

 events from http://elmcity.info    page 9 of 12

Pages fade in, display for 8 seconds, then fade out. There are a million ways to do this, but since I was already exploring IronPython, XAML, and WPF I decided to remix those ingredients. For my own future reference, and for anyone else heading down the same path, here are some notes on what I learned. As always, I welcome suggestions and corrections. I’m still a XAML beginner, and will be very interested to learn about alternative approaches.

The approach I take here is clearly influenced by my own past experience doing web development using dynamic languages. There’s no C# code, no compilation, no Visual Studio. The solution is minimal in the way I strongly prefer for simple projects: a single IronPython script that depends only on IronPython and .NET Framework 3.0.

When developing for the web, I typically build a HTML/JavaScript mockup, view it in a browser, and then consider how to generate that HTML and JavaScript. Here, XAML is the HTML, and a XAML viewer is the browser. The conventional XAML viewer that comes with the Windows SDK is called XAMLPad, but it’s a beefier tool than I needed for this purpose, so I wound up using the more minimal XamlHack.

I started with the contents of a single page:

<Canvas ClipToBounds="True" Background="Black"
  Width="800" Height="600">

<TextBlock x:Name="page1" Canvas.Top="0" Canvas.Left="20"
  Foreground="#FFFFFF" FontSize="36" FontFamily="Arial"
  xml:space="preserve">

<![CDATA[
 06:30 PM open/lap swim  (ymca) 

 07:00 PM Caregiving for Individuals with
   Dementia (unh coop extension)
]]>

</TextBlock>
</Canvas>

I found that text formatting isn’t WPF’s strong suit, so I’m using the XAML equivalent of an HTML <pre> tag to display text that’s preformatted in IronPython.

Next, I added the fade-in and fade-out effects

<Canvas ClipToBounds="True" Background="Black"
  Width="800" Height="600">

<TextBlock.Triggers>
<EventTrigger RoutedEvent="FrameworkElement.Loaded">
  <BeginStoryboard>
    <Storyboard>

     <DoubleAnimation
      BeginTime="0:0:0"
      Storyboard.TargetName="page1"
      Storyboard.TargetProperty="Opacity"
       From="0" To="1" Duration="0:0:1"  /> # 1 sec fade in

     <DoubleAnimation
      BeginTime="0:0:9"                     # wait 8 sec
      Storyboard.TargetName="page1"
      Storyboard.TargetProperty="Opacity"
       From="1" To="0" Duration="0:0:1"  /> # 1 sec fade out

     </Storyboard>
  </BeginStoryboard>
</EventTrigger>
</TextBlock.Triggers>

+ <TextBlock x:Name="page1" Canvas.Top="0" ...>
</Canvas>

I thought it would be possible to chain together a series of these animations, and nest that series inside another animation in order to create the infinite loop that’s required. There may be a way to do that in XAML, but I didn’t find it. So, since I was already planning to generate the XAML — in order to interpolate current event data, plus a variety of attribute values — I went with a generator that produces a series of these pages. That solved chaining, but not looping. To make the sequence loop, I added a second timer/event-handler pair to the IronPython script. The first handler reloads the data once a day. The second handler reloads the XAML at intervals computed according to the number of pages for each day, thus looping the animation.

Next I added XAML elements for the header and footer. The header is static, but the footer has a dynamic page counter so I animated it in the same way as the page.

Next I made templates for all the XAML elements. Here’s the footer template:

template_footer = """<Label x:Name="footer___FOOTER_PAGE_NUM___"
  Canvas.Top="___FOOTER_CANVAS_TOP___" Canvas.Left="___
  FOOTER_CANVAS_LEFT___" Foreground="#FFFFFF" xml:space="preserve"
  FontSize="___FOOTER_FONTSIZE___" FontFamily="Arial" Opacity="0">
           page ___FOOTER_PAGE_NUM___ of ___FOOTER_PAGE_COUNT___
<Label.Triggers>
<EventTrigger RoutedEvent="FrameworkElement.Loaded">
  <BeginStoryboard>
    <Storyboard>
     <DoubleAnimation
      BeginTime="___BEGIN_FADE_IN___"
      Storyboard.TargetName="footer___FOOTER_PAGE_NUM___"
      Storyboard.TargetProperty="Opacity"
       From="0" To="1" Duration="___FADE_DURATION___"  />
     <DoubleAnimation
      BeginTime="___BEGIN_FADE_OUT___"
      Storyboard.TargetName="footer___FOOTER_PAGE_NUM___"
      Storyboard.TargetProperty="Opacity"
       From="1" To="0" Duration="___FADE_DURATION___"  />
     </Storyboard>
  </BeginStoryboard>
</EventTrigger>
</Label.Triggers>
</Label>
"""

The script uses variables that correspond to the uppercase triple-underscore-bracketed names. So, for example:

___FOOTER_CANVAS_TOP___ = 520
___FOOTER_CANVAS_LEFT___ = 10
___FOOTER_FONTSIZE___ = 28

To avoid typing all these names twice in order to interpolate variables into the template, I cheated by defining this pair of Python functions:

def isspecial(key):
  import re
  return re.match('^___.+___$',key) is not None 

def interpolate(localdict,template):
  specialkeys = filter(isspecial,localdict.keys())
  for key in specialkeys:
    exec("""template = template.replace("%s",
      str(localdict['%s']))""" % (key,key))
  return template

Given that setup, here’s the core of the XAML generator:

def create_xaml(raw_text,watch_time,fade_duration):

  ___TITLE_TEXT___ = 'Community Calendar'
  ___BODY_TEXT___ = ''
  ___BODIES_AND_FOOTERS___ = ''
  ___BODY_NUM___ = 0
  ___FOOTER_PAGE_NUM___ = 0
  ___BODY_CANVAS_TOP___ = 0
  ___BODY_CANVAS_LEFT___ = 20
  ___BODY_FONTSIZE___ = 36
  ___TITLE_CANVAS_TOP___ = -30
  ___TITLE_CANVAS_LEFT___ = 200
  ___TITLE_FONTSIZE___ = 34
  ___FOOTER_CANVAS_TOP___ = 520
  ___FOOTER_CANVAS_LEFT___ = 10
  ___FOOTER_FONTSIZE___ = 28
  ___FOOTER_PAGE_COUNT___ = 0
  ___FOOTER_PAGE_NUM___ = 0
  ___BEGIN_FADE_IN___ = ''
  ___BEGIN_FADE_OUT___ = ''
  ___FADE_DURATION___ = ''

  pagecount = 0
  for page in page_iterator(raw_text):
    pagecount += 1
  ___FOOTER_PAGE_COUNT___ = pagecount

  begin_fade_in = 0
  begin_fade_out = begin_fade_in + fade_duration + watch_time

  pagenum = 0

  for page in page_iterator(raw_text):
    pagenum += 1

    ___BODY_TEXT___ = page
    ___BODY_NUM___ = pagenum
    ___FOOTER_PAGE_NUM___ = pagenum
    ___BEGIN_FADE_IN___ = makeMinsSecs(begin_fade_in)
    ___BEGIN_FADE_OUT___ = makeMinsSecs(begin_fade_out)
    ___FADE_DURATION___ = makeMinsSecs(fade_duration)

    body = interpolate(locals(),template_body)

    footer = interpolate(locals(),template_footer)

    ___BODIES_AND_FOOTERS___ += body + footer

    begin_fade_in = begin_fade_out + fade_duration
    begin_fade_out = begin_fade_in + fade_duration
     + watch_time

  xaml = interpolate(locals(),template_xaml)

  return (pagecount,xaml)

I guess I could rely less on XAML code generation and exploit IronPython’s ability to dynamically reach into and modify live .NET objects. That would be the WPF analog to JavaScript DOM-tweaking in the web realm. But this works, it’s easy enough to understand, and it’s handy for debugging purposes to have the generated XAML lying around in a file I can easily inspect.

Finally, here’s the core of the application itself:

class CalendarDisplay(Application):

  def load_xaml(self,filename):
    from System.Windows.Markup import XamlReader
    f = FileStream(filename, FileMode.Open)
    try:
      element = XamlReader.Load(f)
    finally:
      f.Close()
    return element

  def loop_handler(self,sender,args):  # reload XAML

    def update_xaml():
      self.window.Content = self.load_xaml(self.xamlfile)

    self.loop_timer.Dispatcher.Invoke(DispatcherPriority.Normal,
      CallTarget0(update_xaml))

  def day_handler(self,sender,args):     # fetch data, generate XAML

    def update_xaml():
      self.pagecount = calendarToXaml(self.path,self.xamlfile,self.url,
        self.cachefile,self.watch_time,self.fade_duration)
      self.window.Content = self.load_xaml(self.xamlfile)

    self.day_timer.Dispatcher.Invoke(DispatcherPriority.Normal,
      CallTarget0(update_xaml))

  def __init__(self):

    Application.__init__(self)

    self.xamlfile = 'display.xaml'
    self.path = '.'
    self.cachefile = 'last.txt'
    self.url = 'http://elmcity.info/events/todayAsText'
    self.watch_time = 8
    self.fade_duration = 1
    self.pagecount = calendarToXaml(self.path,self.xamlfile,self.url,
      self.cachefile,self.watch_time,self.fade_duration)

    self.window = Window()
    self.window.Content = self.load_xaml(self.xamlfile)
    self.window.WindowStyle = WindowStyle.None
    self.window.WindowState = WindowState.Maximized
    self.window.Topmost = True
    self.window.Cursor = Cursors.None
    self.window.Background = Brushes.Black
    self.window.Foreground = Brushes.White
    self.window.Show()

    self.day_timer = DispatcherTimer()
    self.day_timer.Interval = TimeSpan(24, 0, 0)
    self.day_timer.Tick += self.day_handler
    self.day_timer.Start()

    self.loop_timer = DispatcherTimer()
    interval = self.pagecount * (self.watch_time + self.fade_duration*2)
    self.loop_timer.Interval = TimeSpan(0, 0, interval)
    self.loop_timer.Tick += self.loop_handler
    self.loop_timer.Start()

CalendarDisplay().Run()

Next month marks the tenth anniversary of RFC 2445 (iCalendar), the specification that describes how Internet applications represent and exchange calendar information. The authors of RFC 2445 were Frank Dawson (now with Nokia) and Derik Stenerson (now with Microsoft). I asked both to join me to reflect on the past, present, and future of this key standard. Only Derik was available, and he’s my guest for this week’s ITConversations show.

If you’ve followed my blog you’ll know that I’ve come to regard the ICS files that iCalendar-aware apps create and consume as feeds that could and should form a syndication ecosystem analogous to the RSS ecosystem. So in addition to filling us in on how iCalendar came to be, Derik considers whether the analogy holds water, and concludes that it probably does.

Although iCalendar has been around for a decade, I argue that the confluence of syndication and personal publishing, in the calendar domain, requires three enablers.

First, you need a workable syndication format, and we have that: RSS for blogs, ICS for calendars.

Second, you need what we used to call one-button personal publishing. Bloggers have had that capability for a long time. Calendar users have it too, but it’s emerged relatively recently, and many aren’t aware of it.

Third, you need feed aggregators. These proliferate in blogspace but, I argue, are conspicously absent from calendar space. Services like Eventful and Upcoming produce calendar feeds. But because they do not consume them, they don’t encourage individuals and groups to publish feeds, and to think and act in a syndication-oriented way. I’ve prototyped a calendar aggregator at http://elmcity.info/events/, but the category isn’t yet well-established.

If my analysis is correct, one or more well-known services that both consume and produce calendar feeds would unlock the latent potential of iCalendar and help us jumpstart a calendar syndication ecosystem.

Back in May, This American Life aired a widely-acclaimed show on the mortgage crisis. In The Giant Pool of Money, Alex Blumberg and Adam Davidson pepper their analysis with dialogue from a cast of characters including:

Richard Campbell, ex-Marine, behind on his mortgage: “At one point, my son had $7,000 in a CD and I had to break it. That really hurt.”

Clarence Nathan, who got a $540K second mortgage while working 3 part time not very steady jobs: “I wouldn’t have loaned me the money. And nobody that I know would have loaned me the money. I know guys who are criminals who wouldn’t loan me that and they break your knee-caps.”

Glen Pizzolorusso, just out of college, making $1 million a year selling mortgages to people