Previous month:
July 2010
Next month:
September 2010

August 2010

Jonah Lehrer's Metacognitive Guide to College

HowWeDecide Jonah Lehrer, the 27 year old author of How We Decide, gave the Opening Days convocation keynote at Willamete University last Friday. After being introduced by Willamette president M. Lee Pelton as "a humanist disguised as a neuroscientist", Lehrer offered a fun and fascinating whirlwind tour of neuroscience, psychology and sociology, in the context of a 5-point guide to how to succeed in (and through) college. Having attended several convocations both as a student and a faculty member, I would rank his keynote as one of the best I've ever heard, rivaled only by one I heard in 1986, by Theodor Seuss Geisel (aka Dr. Seuss), when he received an honorary doctorate at the University of Hartford (so he really was a "doctor").

Leading off with a story demonstrating the ephemeral nature of many "great truths" (Oliver Wendel Holmes, Sr., discovering the great "truth" that the world smells like turpentine - specifically, "a strong smell of turpentine prevails throughout" - while on a nitrous oxide-induced hallucinogenic journey), Lehrer assured the new students that they will regularly encounter profound truths and discover new ideas ... few of which will have impact lasting beyond 72 hours, and nearly all of which will be forgotten soon after they finish college. The real value of a college education is learning how to think ... and to promote this process, he offered 5 tips.

Be an outsider is a platform for crowdsourcing research and development that succeeds primarily through the participation of outsiders. Companies post problem descriptions and offer prizes for solutions, and individuals and organizations outside the company submit potential solutions. Lehrer quoted a Harvard Business School study reporting that 60% of the posted problems are solved within 6 months, and that the key to solutions is being on the outside, i.e., being able to look at the problem from an outsider's perspective.

I'm not sure which study he is referencing (I haven't read his book yet), but I did find a related report on an Innovation Network conference:

InnoCentive now boasts 175,000 "solvers" from more than 200 countries around the world. About 90% are individuals, 10% are organizations and 60% have masters degrees or PhDs. Last year, nearly 50% of the "challenges" posted on InnoCentive's web site generated a solution that was put to use.

Academics who polled InnoCentive's winning solvers discovered something "both startling and intuitively obvious," said Spradlin. "What they found was that typically ... the background of the solver who solved the problem" was "no less than six disciplines away" from the subject area in which the problem emerged. "What that means is, if all the Stanford PhDs in your chemistry lab could have solved the problem, they would have solved it already."

Lehrer reported that English poet Samuel Taylor Coleridge used to tell people he attended public lectures on chemistry in London "to improve my stock of metaphors", and encouraged students to take at least one class each semester outside of their field ... and "don't be afraid to be the lonely poet in chem class".

Learn how to relax

image from Lehrer described a study on people solving compound remote associates problems, for which Lehrer suggested the evocative acronym "CRAP". Another acronym, "RAT" (remote associates test), is more commonly associated with these kinds of problems - often posed on the Sunday Puzzle on NPR - in which three words are presented and the problem is to find a fourth word that relates to all of them (e.g., given the problem "broken, clear, eye", the solution is "glass"). The study revealed that the "flash of insight" or "Aha!" moment that occurs immediately before a solution can be reliably detected via functional magnetic resonance imaging (fMRI) and electroencephalography (EEG), and the alpha wave pattern closely resembles that of someone who has experience in meditation, i.e., someone who is able to achieve states of deep relaxation.

Contrary to the intuition many of us have when faced with a hard problem, which is to focus on the problem as hard as we can (I imagine this is why they are called "hard problems"), the solution in many cases is to simply relax and temporarily turn our attention to other things, and allow the solution to emerge more organically. I was reminded of one of my favorite lines of poetry, by Wallace Stevens:

Perhaps the truth depends on a walk around a lake.

Another observation by Lehrer - the brain knows more than you know, you just have to listen - reminded me of the way yet another poet, David Whyte, describes poetry:

Poetry is the art of overhearing yourself say things you didn't know you knew.

But I digress. Shifting from poetry to technology - and back to Lehrer's speech - Lehrer suggested that one of the most effective ways of listening to what you know is to turn off the gadgets that constantly inundate us with what others are saying ... reminding me of what Sherry Turkle, Kathy Sierra, James Surowiecki, Malcolm Gladwell, James Ogilvy, Dan Oestreich and other great thinkers have said about self-reflection vs. self-expression, and the recent New York Times article on digital devices deprive brain of needed downtime.

Make friends with lots of different people

ConsequentialStrangers Lehrer described the self-similarity principle (or perhaps homophily) as a natural tendency to associate with people who are like us (and avoid people who are not like us), and suggested that students guard against this tendency. A study by sociologist Martin Ruef and his colleagues at Princeton, in which they interviewed 600 entrepreneurs, revealed that the entrepreneurs with the highest informational entropy (i.e., most diverse social networks) were the most successful, and that the propensity to strike up conversation with potentially consequential strangers was a key indicator of this quality. The researchers estimate that entrepreneurs with highly entropic networks were 3 times more innovative than those with low entropy networks (though innovation is a notoriously difficult concept to measure).

College is a great place to forge new connections with a broad range of people, and so Lehrer encouraged students to take advantage of the opportunity to diversify their social networks ... which will seve them well long after they've forgotten all (or most of) the facts they will have learned while in school.

Don't eat the marshmallow

image from Another variation on the theme of intent focus vs. relaxation - or, at least, distraction - was illuminated through the story of the marshmallow task, which Lehrer wrote about in a New Yorker article on the secret of self control last May. Stanford psychology professor Walter Mischel conducted experiments with four year olds at the Bing Nursery School, including one named Carolyn, to explore delayed gratification:

Carolyn was asked to sit down in the chair and pick a treat from a tray of marshmallows, cookies, and pretzel sticks. Carolyn chose the marshmallow. ... A researcher then made Carolyn an offer: she could either eat one marshmallow right away or, if she was willing to wait while he stepped out for a few minutes, she could have two marshmallows when he returned. He said that if she rang a bell on the desk while he was away he would come running back, and she could eat one marshmallow but would forfeit the second. Then he left the room [for about 15 minutes].

Only 30% of the children were able to delay gratification for the full 15 minutes; the average delay of gratification was about 2 minutes. 13 years later, Mischel conducted extensive followup surveys to discover how the 600+ children had fared. The high delayers - those who were able to distract themselves for the full 15 minutes - averaged 200 points higher on the SAT, on average, than the low delayers - those who were unable to shift their attention to anything but the marshmallow, and succumbed to temptation within 30 seconds.

Lehrer instructed the students that "your task for the next four years is to learn how to control your attention. You control the spotlight" - use it wisely.


BeingWrongBook Elaborating on a theme invoked by Dean Darlene Moore during the opening remarks to the event - in which she emphasized the primacy of the journey over the destination - Lehrer invited students to fully appreciate the experience of a college education. Highlighting the importance of embracing wrongology, Lehrer offered a great anecdote:

You get to share your opinion on Hamlet, and write long essays about how Plato, the guy who blew your mind last week, was actually wrong about everything.

In my own experience as a philosophy major years ago (and continuing ever since), education is about learning things, and then unlearning things; discovering a great truth, and then discovering that its opposite is [also] true. I can understand the appeal of fundamentalism, in clinging tenaciously to beliefs no matter what facts may present themselves, especially as fears, uncertainties and doubts are promulgated by those who would deign to decide for us, but I don't think we can learn much when we are not willing to be in the question(s).

image from upload.wikimedia.orgSpeaking of questions, during the question & answer period following his talk, my favorite question was by a student who asked how Lehrer figures out which questions to ask (or pursue). He answered that he wrote a book about decisions primarily because he is pathologically indecisive, and generally tends to begin with his own frustrations ... mirroring my own tendency toward what I like to call irritation-based research ... or what Eric Raymond, author of The Cathedral and the Bazaar, describes in the context of open source programming:

Every good work of software starts by scratching a developer's personal itch.

In closing, I want to acknowledge that I have not yet read Lehrer's book, How We Decide, but as I noted in my earlier post on the warm welcome we enjoyed throughout Willamette Opening Days, my daughter, Meg, read the book over the summer, and after his speech she told me that many of the examples are covered more extensively in the book, which is next on my stack of "to-reads".

image from Finally, I want to loop back to some introductory remarks made by President Pelton, in which he quoted E. O. Wilson, the multidisciplinary scientist sociobiologist who contributed much to our understanding of ant colonies (and other societies and systems):

We are drowning in information, while starving for wisdom. The world henceforth will be run by synthesizers, people able to put together the right information at the right time, think critically about it, and make important choices wisely.

Lehrer, like Wilson, is clearly a great synthesizer, and I hope his convocation keynote - and the subsequent scope of a liberal arts education at Willamette - will help inspire a future generation of synthesizers, critical thinkers and wise decision-makers.

A warm welcome at Willamette University Opening Days

image from We brought our daughter down to Willamette University this week and enjoyed a warm welcome during their Opening Days orientation program. I had written about our short tour of small colleges in the Pacific Northwest last March, which included Willamette and several other schools she was considering. Meg eventually applied to and was admitted to several very good schools, some of which included attractive scholarship offers. The overall package of education quality, campus life and scholarship offered by Willamette seemed to offer the best fit for her aspirations, and our experience at opening days only reinforced the sense that she had made the right decision.

Bearcat move-in squad TIUA move-in squad When we arrived at the Salem, OR, campus after our 4-hour drive Thursday morning, we were pleasantly surprised to be greeted by a group of Willamette Bearcats football players and students from Tokyo International University of America - which is affiliated and collocated with Willamette - who offered to carry Meg's stuff up to her dorm room. The football players have traditionally helped new students who are on the team move in; this year, Scott (second from left in photo to the left) said he thought it would be a nice gesture - very much in keeping with the traditions of community service at Willamette - to help all students move in this year. TIUA also has a strong tradition of community service. A collection of students from both groups came right up to the van, each grabbing an armful of stuff, and had all of Meg's things outside her room in under 5 minutes. This was the third time we've been to the campus, and each time we had a positive experience, but I have to say that this initial greeting made a powerfully positive impression on us.

image from After lunch at Goudy Commons, we attended the Welcome Program for New Students and Families, which included presentations by Willamette President M. Lee Pelton, VP of Enrollment & Financial Aid Madeleine Rhyneer and Opening Days Coordinator Emma Larkins. Among the things we learned was that this year was the first time that Willamette admitted fewer than 50% of its applicants; other statistics revealed on the welcome page for the class of 2014 include a median SAT score of 1870 (50 points higher than last year's class), a median high school GPA of 3.79, and over half (51%) ranked in the top 10% of their high school graduating class (a 10% increase from last year). We also learned that many of the Opening Days coordinators are pretty good dancers.

The students and their families were then separated, and Amy and I attended a session on Campus Life at Willamette. I had never heard of the Family Educational Rights and Privacy Act (FERPA) before - even though it was enacted in 1974, and so was in force when I attended college - but we learned that a student must sign a waiver for the university to release any information about any aspect of the student's life and work at the school. We also learned about a range of counseling and other support services available to students at the college.

Having been deeply disturbed by a recent NPR series on campus rape - an estimated 20% of women are sexually assaulted at some point during their college career, and even at the most progressive institutions (with respect to this issue) only 10-25% of men found guilty of sexual assault are expelled - I asked about the prevalence and policies regarding sexual assault on campus. Margaret Trout, Director of the Bishop Wellness Center on campus, said that the prevalance of non-stranger sexual assault at Willamette is consistent with the national average, and that a survivor can choose whether to press charges with the Salem police or through the Willamette judicial system; if they choose the campus judicial system, and the perpetrator is found guilty, that person will be expelled. She also said that the campus has trained student volunteers who serve as sexual assault response allies, and this has increased both the reporting and effective response to sexual assault on campus. On the one hand, I was reluctant to raise the issue in the session (or in this post), but on the other hand, I think it is very important that parents - and students - be aware of how prevalent this problem is, and how ineffective many schools are in dealing with it.

After the campus life session, parents and students enjoyed a nice picnic dinner while being entertained by the Los Palmeros Mariachi band. We discovered that a friend and former colleague from the Seattle area also has a son who is an entering freshman at Willamette, and that the son's best friend - whose older brother is a junior at Willamette - is also a freshman there. It was nice to reconnect with local friends and to discover that there is a tradition of siblings attending the same school (not that we have any preconceived notions that what is appealing to Meg will be appealing to Evan).

We then attended an evening session on Residence Life at Willamette, where Resident Assistants were on stage to answer any questions parents or students might have about living on campus. Willamette does not have any campus-wide policy with respect to "quiet hours", instead imposing 24-hour "courtesy hours", relying upon the discretion and judgment of the students to respect their peers, but also allowing individual dorms (and residences) to dictate specific rules if individual discretion and judgment do not match residents' expectations of courtesy. Other topics that were emphasized were the importance of locking doors, windows and bikes (the exclusive use of U-shaped Kryptonite locks was encouraged both in this session and the earlier session on Campus Life).

We spent the night at the Grand Hotel in downtown Salem - whose #1 rating on TripAdvisor is well-deserved - and enjoyed such a restful sleep that we missed the early morning sessions on the second day. However, we did attend the Opening Convocation at 10:30, with a keynote by Jonah Lehrer. That was such an inspiring experience in and of itself, I'm going to split that off into a separate blog post [update: I've posted my notes on Jonah Lehrer's Metacognitive Guide to College]. For this post, I'll simply note that I was also inspired by Dean Marlene Moore's opening remarks during the convocation, in which she invoked a metaphor of "curriculum as conversation", encouraged both curiosity and confusion (the latter being one of the best routes to eventual clarity), and emphasized the importance of the journey (the college experience) over the destination (the degree).

I'll finish this post by observing the synchronicity of Meg having bought Lehrer's most recent book, How We Decide, in June, before the convocation speaker was announced, and so the selection of Jonah Lehrer as the convocation speaker just adds more corroboration (for me) that Willamette was, indeed, the right decision.

Hadoop Day in Seattle: Hadoop, Cascading, Hive and Pig

image from I attended Hadoop Day - a community event to spread the love of Hadoop and Big Data - at Amazon's Pac-Med building in Seattle a week ago. I missed the morning session of the event, but recently became better acquainted with some of the dimensions of this space via the excellent overview and analysis by Mike Loukides at O'Reilly Radar, What Is Data Science? The afternoon "Introductory Track" included presentations about a number of tools for processing large data sets - Hadoop, Cascading, Hive and Pig - by large and small companies involved with big data - KarmaSphere, Drawn to Scale, Facebook and Yahoo. The session was intended as a hands-on learning opportunity, but due in part to poor network connectivity, it ended up being mostly an eyes- and ears-on educational event (but still very worthwhile).

karmasphere Abe Taha, VP of Engineering for Karmasphere, started the afternoon session with 0-60: Hadoop Development in 60 Minutes or Less (slides embedded below), which offered a great general introduction to Hadoop, a preview of the other tools that would be presented in later sessions (from different levels of the Hadoop stack) and an appropriately scaled (i.e., relatively brief and informative) demonstration of the Karmasphere Studio tool.

Abe led off with the motivation behind Hadoop: the need for a scalable tool for discovering insights (or at least patterns) in ever-increasing collections of data, such as logs of web site traffic. Hadoop embodies the MapReduce paradigm in which data is represented as records or tuples, and computing processes can be broken down into mapping - in which some function is computed over a subset of tuples - and reducing -  in which the results of the applications of the mapping function to different subsets are then combined. The power of Hadoop comes in being able to farm out the functions and different data subsets across a cluster of computers, potentially increasing the speed of deriving a result.

image from Simple examples were offered to illustrate how Hadoop works, e.g., computing the maximum of a set of numbers, adding a set of numbers, and counting the occurrences of words in a large text or collection of texts (e.g., The Complete Works of William Shakespeare). After reviewing how these data sets might be represented in Hadoop, Abe provided some Java code to illustrate how the map and reduce functions could be implemented to process them (these code segments are included in the slides). Although the poor network connectivity precluded trying running the code during the session, the clear presentation and simple examples left a relative newcomer like me with the sense that "I can do this" (which I believe was the main objective for the day).

Over half of Abe's slides were on Karmasphere Studio (starting around slide #26 (out of 66)), and the way it can help address some of the problems with overhead in Hadoop, particularly with respect to allowing debugging, prototyping and testing without having to deploy to a cluster of computers. However, only about a quarter of the hour-long presentation was devoted to the tool, and given that the Community Edition of Karmasphere Studio is available for free, I thought he achieved the right balance between covering Hadoop fundamantals as well as a tool for using Hadoop.

image from Next up was Bradford Stephens, founder of Drawn to Scale and organizer of the event, who presented an Introduction to Cascading. Cascading is a layer on top of Hadoop that allows users to think and work at a higher level of abstraction, focusing on workflows rather than mapping and reducing functions. Cascading offers a collection of operations - functions, filters and aggregators - that can be used in conjunction with any Java Virtual Machine-based language. Bradford showed a 15-line sample of Cascading code to process apache web server logs, and an equivalent 200 line Java program to do the same thing.

Bradford offered the most interactive exercise of the afternoon, showing us some Cascading code to process New York Stock Exchange closing prices, and inviting us to help him write the code that would find the symbol and price of the stock with the highest closing price for each of the days represented in the dataset. I cannot find the slides for Bradford's talk, but the code and data he used in the examples are available at the main Hadoop Day site.

image from After a break, Ning Zhang, a software engineer at Facebook, presented an introduction to Hive entitled a Petabyte Scale Data Warehousing System on Hadoop (slides embedded below).

Facebook_logo Ning presented some statistics about Facebook I'd heard or read elsewhere - e.g., 500M monthly active users, 130 friends per user (on average) - along with several I had not known before:

  • 250 million daily active users
  • 160 million active objects (groups, events, pages)
  • 60 object (group/event/page) connections per user (on average)
  • 500 billion minutes per month spent on the site across all users
  • 25 billion content items are shared per month across all users
  • 70 content items created per user per month (on average)
  • 200 GB of data/day was being generated on the site in March 2008
  • 12+ TB of data/day was being generated by end of 2009, growing by 8x annually

In addition to the increasing demands of users, Facebook application developers and advertisers want feedback on their apps and ads. Facebook decided against using closed, proprietary systems due to issues of cost, scalability and length of the development and release cycles. They considered using Hadoop, but wanted something that provided a higher level of abstraction and used the kinds of schemas traditionally provided in relational database management systems (RDBMS). Hive provides the capability to express operations in SQL and have them translated into the MapReduce framework, and provides extensive support for optimizations ... a dimension that is increasingly important for a company with increasingly big data needs.

image from Alan Gates, a software architect on the Yahoo! grid team, led off his talk on Pig, Making Hadoop Easy (slides embedded below) with a motivating example, showing how a 200 (?) line program to find the top 5 sites visited by 18-25 year olds in Java using Hadoop directly could be written as a 10 line program in Pig Latin. Pig represents a middle way between straight Hadoop and the higher level abstractions provided by Cascading and Hive, providing the capability to program in a higher level scripting language (i.e., higher level than Java) while still being able to define elements procedurally (vs. the declarative definitions typical of SQL-oriented frameworks).

Yahoo_logo Alan recently wrote a blog post on Pig and Hive at Yahoo! in which he delves more deeply into the similarities and differences between the two frameworks, both of which have their place(s) in the realm of data warehousing. Data processing typically involves three phases: data collection, data preparation, and data presentation. Pig is particularly well suited to three tasks involved in the data preparation phase (aka Extract Transform Load (ETL) or data factory):

According to Alan, Hive is well suited to the data presentation phase, in which business intelligence analysis and ad hoc queries may be better accommodated by a language that directly supports SQL. It seems to me that an argument could be made that these tasks might also be categorized as research, though perhaps the differentiation between phases lies more in the types of questions one might most easily be able to ask (and answer). In any case, the data and code used by Alan in his talk are also available on the Hadoop Day site.

Alan Gates at Hadoop Day In addition to interesting presentations, there were some other interesting things I noted about the group at the event. I would estimate that 95% of the attendees were male - much higher than the events I typically attend, which focus more on human-computer interaction and social computing - and the proportion of Macs was much lower than other events I typically attend - perhaps 30%, with nearly 50% of the laptops I saw being Thinkpads. The presentations and presenters were great, as was the view and the food; the only downside was the poor wireless connectivity ... which was somewhat surprising, given the site (Amazon), but was probably due to the need to rig an ad-hoc network outside the firewall just for the event.

All in all, it was a very worthwhile day, and I'm grateful to the organizers, the sponsors - Amazon Web Services, Fabric Worldwide, Cloudera and Karmasphere - and all the presenters for putting the event together. There is already some talk about holding another Eastside Networking Event, a technology-oriented event with several representatives from the local big data community (Amazon Web Services, Microsoft Windows Azure and Facebook [Seattle]) that I wrote about in my last post. I don't know whether there will be future day-long Hadoop events in Seattle, but there are monthly Hadoop meetups in Seattle which are also organized by Bradford; the next meeting of the group is this Wednesday.

Wine, Cheese, Technology and Jobs in Seattle

The Eastside Networking Event last night, organized by Andrew Vest, included an interesting mix of wines, cheeses and Seattle area technology companies looking to hire people - primarily, but not exclusively, engineers. At one point, one of the speakers asked how many people in the room were engineers, and only about one quarter or so of the 400+ people in the audience raised their hands, so I'm not sure how well expectations were met among the sponsors and attendees of the event, but I enjoyed meeting interesting people, learning more about the companies and trying some new wines.

image from Mike Whitmore, of Fresh Consulting, talked about the increasing ways that technology is permeating our lives. The company, which integrates business, technology and design, makes extensive use of Amazon's Mechanical Turk and oDesk for outsourcing its work, using 3,000 people for micro-tasks and mini-tasks from these two services ... most of whom, I imagine, do not work in the greater Seattle area. One of the sites the company has created is, which allows users to rate web apps, and then provides capabilities to compare, rank and filter those reviews. Mike mentioned some kind of prize(s) associated with using the site, but I wasn't clear on the details and has kindly clarified the Kindle Contest in a comment below:

  1. Go to and click on the Community tab.
  2. Click the green Quick Signup button.
  3. Follow the process to Create your Account by August 31st.

All who join the CloudSurfer Community by August 31st are eligible for a drawing for a Kindle 3G.

image from My favorite example of new technologies from Mike's talk was Kickbee, a device that can be attached to the belly of a pregnant woman that tweets every time the fetus kicks. There were several articles (including the Gizmodo article I link to above) about this device in December 2008, but the official Kickbee site, and the project page in the portfolio of the device's creator, Corey Menscher (when he was a student at NYU's Interactive Telecommunications Program), are both currently marked by "This site may harm your computer" warnings in Google. @kickbee reminds me of @PiMPY3WASH, a device connected to a Maytag washing machine that tweets when its load is done [video] ... which seems somewhat less risky than attaching sensors and transmitters to the belly of a pregnant woman. In any case, I agree with Mike that there are many interesting developments in the increasing array of activity streams that are connected to the web, and many exciting opportunities to create useful new services based on the greater awareness and interaction capabilities these offer.

image from Next up was Werner Vogels, CTO of Amazon, who was touting Amazon Web Services. Werner spoke of how AWS has drastically reduced the time and costs of developing enterprise software - from 2 years down to 3-6 months - and increases agility by enabling scaling up and down based on actual (vs. anticipated) demand. Werner claimed that 7 of the top 10 Facebook apps are games (I'm not sure how "top" is measured), all of which run on AWS, and mentioned several non-game web services that I know (and love), and included one that I hadn't heard of before, but will definitely use next time I plan air travel: hipmunk. One of my favorite phrases from the event was Werner's reference to the way AWS helps during that one exponential moment when your web site gets a large spike of visitors (I've always thought of this as getting Boing-Boinged, but "exponential moment" is perhaps more general).

WindowsPhone7 Charlie Kindel, general manager for user experience with the Microsoft Windows Phone 7 team emphasized the game-changing approach that Microsoft is taking to this new platform, i.e., how they are changing their game as they seek to compete more effectively in the smart phone market. The Windows Phone 7 is designed to offer integrated experiences, combining elements of the personal and the professional dimensions of our lives through the incorporation of productivity tools as well as photos, videos, music and games (at various points, Charlie talked about "the Zune phone" and "the Xbox phone"). The live demo - using an app that replicated the phone screen on a laptop screen - was pretty impressive, highlighting the "live tiles" interface and the integration with mail, maps and the calendar. My favorite feature was the button on the calendar app that allows a user to automatically generate an "I'll be late" email to the organizer or all attendees of an event. In addition to the user experience, Charlie emphasized the Windows Phone developer experience, claiming at one point that for someone with any software development experience [presumably with Microsoft tools and languages], it can take as little as 2-3 minutes to get a Windows Phone 7 app up and running (!).

image from Barbara Evans, aka Seattle Wine Gal, and Community Manager of Thinkspace, promotes social media for and about wine and wineries, and talked about how she started out with no qualifications, but lots of passion & drive to learn about the social online space. I first met Barbara at the afterparty for a daylong conference in Seattle this past winter, for which she had assembled one of the best selections of wineries to pour at a tasting I've encountered in Seattle. I don't know about her claim of "no qualifications": she has a degree in social anthropology, great taste in wines (and wineries), and I suspect that the Boopsie Effect - wherein attractive women suffer disadvantages due to their appearance in certain professions - does not apply to social media (in fact, I imagine it is more the opposite). She attempted to show a video of how to drink wine in the shower during the presentation, but it did not work; having watched it today, despite the images that might come to mind regarding a shower scene, I can say that it is not NSFW.

image from Bill Baxter, CTO of Cozi, which offers a set of applications to keep busy families organized, offered a blend of "prognosticating and bitching" about the state of web and mobile apps. Citing an "unstoppable juggernaut named Apple", he warned that the current rate of platform proliferation - iOS, Facebook, Windows Mobile 7, WIndows 7/8 (desktop), Android, Safari, Chrome, Internet Explorer 9, HTML5, Blackberry OS, Yahoo Connected TV, Google TV - was unsustainable. He predicted that HTML5/CSS3/JavaScript will be the likely winner, but meanwhile, developers will be forced to create native clients for many plaforms, and so skills in many languages and tools - Objective-C, Java, HTML/CSS/JavaScript, as well as backend skills in database and web services - will continue to be invaluable for the forseeable future. Interestingly (to me), Bill's Twitter account has the fewest followers (3), followees (7) and tweets (2) of any of the speakers, and I can't find a LinkedIn profile either - though @CoziFamily has plenty of followers, followees and tweets - leading me to wonder if social media restraint is a strategy intended to promote the simplification and organization of his [family] life.

Windows_azure_logo Bharat Shyam, General Manager of Microsoft Windows Azure, has no Twitter account (that I can find), which is consistent with his opening statement about how little he typically tends to enjoy networking. Just as Windows Phone 7 is a relative latecomer to the smart phone game, Windows Azure is a relative latecomer to the cloud computing game, having been unveiled in January of this year (although in describing Amazon Web Services, which was an early pioneer in this area 4 years ago, Werner Vogels described the field as still feeling like "day 1"). Bharat emphasized the powerful and popular tools available to support developers on the Azure platform, e.g., Visual Studio for Windows Azure, and said there were already 10,000 Windows Azure developers, and 10,000 Windows Azure customers, a number which is "growing at a rapid clip".

EVenues_logo Nic Peterson, CEO of eVenues, whose Twitter account also exhibits restraint, gave a demo of their "marketplace for underutilized space", showing how a user can search for and reserve an unused conference room - or desk, classroom or event space - owned by another organization based on hourly or daily rates. Nic claimed that the "under 50" [person] meeting space market is a $2.5B business, and noted that hotels charge more than 10 times the eVenue rates (and having organized numerous events, I can attest to high rates in hotels). He said they soon plan to release a backend tool for space owners to more easily manage the availability of their spaces on eVenues, and I found myself wondering about potential partnerships with catering and transportation companies.

Facebook_logo Ari Steinberg, manager of the new Facebook office in Seattle, showed a bunch of photos of the new office (in Pioneer Square), and told us about another networking event / party they are holding at their office next week ... which, unfortunately, is already full. In their quest to ensure that the party is populated primarily by clever engineers, they presented a puzzle as a gatekeeper to the event. Engineers who are late to the party (and/or puzzle), may increase their chances for being invited to future Facebook parties - or perhaps even employment - by doing clever things with the Facebook Graph API, which provides access to 20 types of Facebook objects, including people, events, groups, friends, photos, videos, likes, notes, links and - after yesterday's announcement - places.

Gist_logo T.A. McCann, CEO of Gist ("know more about who you know"), was up last, and started off with the contextually appropriate insight that "the key to social networking is wine". Gist was created to deal with the information overload some of us experience with too many inboxes, social networking connections. Gist combines Outlook, Gmail, LinkedIn, Facebook, Twitter, news, blogs, photos, RSS feeds and contact info into a single stream (for each person), making it easier to learn what's new on a person-by-person basis, regardless of the preferred platform(s) through which that person prefers to express him- or herself. According to T.A., Gist already has more profiles than LinkedIn (!), and before ending his talk - and with it, the presentation portion of the event - he noted the company has 20 employees in Pioneer Square, and "we are always hiring".

Last, but not least, I wanted to say a few words about the wine. Tasting tables were setup during the first speaker session, and the wine was flowing by the time of the first break. There was also a table for Maker's Mark Kentucky Bourbon (on the rocks or in a punch made with Reed's Ginger Beer). The wineries represented - and the wines poured - were

  • Bartholomew Winery poured a 2007 Orsa (69% Syrah, 18% Mourvedre, 13% Grenache), 2007 Reciprocity (50% Cabernet Sauvignon, 50% Carmenere) - my favorite of the night - and 2007 Cuvee Rouge (a Cabernet / Merlot blend)
  • Celaeno Winery poured a Gewurtzraminer - the most interesting wine of the night (a prominent smokey flavor, almost like drinking bacon) - a Syrah and a Cabernet Sauvignon
  • Mount Baker Vineyards poured a Semillon / Sauvignon Blanc blend, a Cabernet Franc and a Syrah (I did not make careful notes of the origins or vintages)
  • 509 Wines poured a Viognier - my second favorite wine - a Cabernet and a Syrah

The facts and opinions surrounding the Proposition 8 ruling

image from I was elated by U.S. District Court Judge Vaughan Walker's ruling agaist California's Propostion 8 last week. The 136-page ruling, overturning the state's ban on same-sex marriage, renews my hope for full civil rights for all of us's, including lesbian, gay, bi-sexual and transgender (LGBT) people. Over the weekend, I have read some interesting facts and opinions about the facts and opinions presented in the case. I also watched - and was moved in different ways by - Boys Don't Cry and The Kids are All Right, but I will stick to reviewing news articles rather than movies in the context of this post.

Dahlia Lithwick wrote a brilliant article on A Brilliant Ruling that appeared in Slate on Wednesday, in which she reports on some of the facts and opinions that were expressed by witnesses for the plaintiffs (two same-sex couples) and the defense (the state of California and opponents of same-sex marriage) in the U.S. District Court case. After noting the likelihood of an appeal of the ruling proceeding all the way up to the Supreme Court, the carefully crafted 15 citations to 2 Supreme Court decisions written by Justice Anthony Kennedy, and commenting on the imbalance in preparation and presentation of expert testimony between the two sides in the trial, she highlights some specific findings articulated by Judge Walker that I find difficult to dispute (and I hope that Justice Kennedy and his colleagues will encounter similar difficulties in disputing the findings):

Then come the elaborate "findings of fact"—and recall that appellate courts must defer far more to a judge's findings of fact than conclusions of law. Here is where Judge Walker knits together the trial evidence, to the data, to the nerves at the very base of Justice Kennedy's brain. Among his most notable determinations of fact, Walker finds: states have long discriminated in matters of who can marry; marital status affects immigration, citizenship, tax policy, property and inheritance rules, and benefits programs; that individuals do not choose their own sexual orientation; California law encourages gay couples to become parents; domestic partnership is a second-class legal status; permitting same-sex couples to marry does not affect the number of opposite-sex couples who marry, divorce, cohabit, or otherwise screw around. He found that it benefits the children of gay parents to have them be married and that the gender of a child's parent is not a factor in a child's adjustment. He found that Prop 8 puts the force of law behind a social stigma and that the entirety of the Prop 8 campaign relied on instilling fears that children exposed to the concept of same-sex marriage may become gay. (Brand-new data show that the needle only really moved in favor of the Prop 8 camp when parents of young children came out in force against gay marriage in the 11th hour of the campaign.) He found that stereotypes targeting gays and lesbians have resulted in terrible disadvantages for them and that the Prop 8 campaign traded on those stereotypes.

image from After reading this article, I decided to download and read the full text of the ruling in Perry v Schwarzenegger [PDF] for myself. [Update: HTML5 version of the Prop 8 ruling on Scribd embedded at bottom.] Of particular interest, to me, was the testimony of one of the only two expert witnesses called by the proponents of Proposition 8 called in the case: David Blankenhorn, the founder and president of the Institute for American Values and author of The Future of Marriage.

Here are some particularly poignant passages regarding Blankenhorn's testimony, which the court concluded constitutes "inadmissible opinion testimony that should be given essentially no weight" (with a few links to items referenced directly or indirectly inserted):

During trial, Blankenhorn was presented with a study that posed an empirical question whether permitting marriage or civil unions for same-sex couples would lead to the manifestations Blankenhorn described as indicative of deinstitutionalization. After reviewing and analyzing available evidence, the study concludes that “laws permitting same-sex marriage or civil unions have no adverse effect on marriage, divorce, and abortion rates, the percent of children born out of wedlock, or the percent of households with children under 18 headed by women.” PX2898 (Laura Langbein & Mark A Yost, Jr, Same-Sex Marriage and Negative Externalities, 90 Soc Sci Q 2 (June 2009) at 305-306). Blankenhorn had not seen the study before trial and was thus unfamiliar with its methods and conclusions. Nevertheless, Blankenhorn dismissed the study and its results, reasoning that its authors “think that [the conclusion is] so self-evident that anybody who has an opposing point of view is not a rational person.”

I added emphasis to the last sentence, as to highlight the irony of one purported expert summarily dismissing the work of other purported experts while accusing them of, in effect, summarily dismissing other points of view that differ from their own ... reminding me of another article I read recently about a study that suggests what you say about others says a lot about you. Judge Walker similalry noted that Blankenhorn "failed to consider evidence contrary to his view in presenting his testimony".

In my reading of the case, it appears that Blankenhorn, if anything, offered testimony that tended to support the claims of the plaintiffs (the same-sex couples), against which he had been invited to testify:

Blankenhorn’s concern that same-sex marriage poses a threat to the institution of marriage is further undermined by his testimony that same-sex marriage and opposite-sex marriage operate almost identically. During cross-examination, Blankenhorn was shown a report produced by his Institute in 2000 explaining the six dimensions of marriage: (1) legal contract; (2) financial partnership; (3) sacred promise; (4) sexual union; (5) personal bond; and (6) family-making bond. PX2879 (Coalition for Marriage, Family and Couples Education, et al, The Marriage Movement: A Statement of Principles (Institute for American Values 2000)). Blankenhorn agreed that same-sex marriages and opposite-sex marriages would be identical across these six dimensions. Tr 2913:8-2916:18. When referring to the sixth dimension, a family-making bond, Blankenhorn agreed that same-sex couples could “raise” children. Tr 2916:17.
Blankenhorn testified on cross-examination that studies show children of adoptive parents do as well or better than children of biological parents [possibly referring to Farr, R. H., Forssell, S. L., & Patterson, C. J. (2010). Parenting and child development in adoptive families: Does parental sexual orientation matter? Applied Developmental Science, 10, 164-178]. Tr 2794:12-2795:5. Blankenhorn agreed that children raised by same-sex couples would benefit if their parents were permitted to marry. Tr 2803:6-15. Blankenhorn also testified he wrote and agrees with the statement “I believe that today the principle of equal human dignity must apply to gay and lesbian persons. In that sense, insofar as we are a nation founded on this principle, we would be more American on the day we permitted same-sex marriage than we were the day before.” DIX0956 at 2; Tr 2805:6-2806:1.
Blankenhorn stated he opposes marriage for same-sex couples because it will weaken the institution of marriage, despite his recognition that at least thirteen positive consequences would flow from state recognition of marriage for same-sex couples, including: (1) by increasing the number of married couples who might be interested in adoption and foster care, same-sex marriage might well lead to fewer children growing up in state institutions and more children growing up in loving adoptive and foster families; and (2) same-sex marriage would signify greater social acceptance of homosexual love and the worth and validity of same-sex intimate relationships. Tr 2839:16-2842:25; 2847:1-2848:3; DIX0956 at 203-205.

Dana Mack, who co-authored an earlier book with Blankenhorn in 2001, The Book of Marriage: The Wisest Answers to the Toughest Questions, wrote a spirited defense of Blankenhorn's testimony and views in her Wall Street Journal opinion piece on Friday, Now What for Marriage?. However, like her colleague, Ms. Mack appears to exhibit the same unwillingness to consider alternate perspectives:

there is simply no other way to view the age-old, universal institution of marriage than as rooted in the biological family

Also consistent with the conduct of her colleague, she makes claims without offering much scientific evidence; and while I don't mean to imply that a newspaper opinion article ought to have the exact same standards as courtroom testimony, I do believe that opinions with factual corroboration generally carry more weight ... at least among those who care about facts.

In this case, Mack claims that "there is a great deal of social-science evidence connecting marriage and the active engagement of two biological parents with child well-being", and yet only makes reference to a single statement made by anthropologist Bronislaw Molinowski without any reference to supporting studies he or others may have conducted.

I would argue that, as was the case with her colleague, she raises issues that may unintentionally serve to further support the plaintiff's case. After suggesting that the institution of marriage's "common denominator across time and cultures has been its dedication to the offices of reproduction", she goes on to report that "A recent Pew analysis of 2008 census data showed that only just over 40% of Americans consider children fundamental to marriage", a decline from 65% who expressed that view in 1990. I believe she quotes this statistic to bolster her and Blankenthorn's supposition that marriage is being "de-institutionalized"; in my reading, it suggests that the institution of marriage is being progressively re-defined rather than being undermined.

image from

The report I believe she is referring to, Childlessness Up Among All Women; Down Among Women with Advanced Degrees, shows that the proportion of all women who end their childbearing years without bearing children has risen from 10% in 1976 to 18% in 2008; among women who have ever been married, that rate is 13%. If procreation is an essential element of a "successful" or "legal" marriage, I don't know how Mack, Blankenhorn and other opponents of same-sex marriage regard childless marriages, but I have several married friends who are childless - or childfree - by choice, and I don't believe that they see their marriages as any less legitimate than childful marriages. I certainly don't, nor do I see same-sex marriages as any less legitimate than heterosexual marriages ... and I hope the growing preponderence of facts supporting the positive aspects of same-sex marriages will help influence more opinions so that we can move beyond this controversy and devote more attention to other social, economic and political problems in dire need of resolution.

Update: Just found the full text of the ruling on Scribd; embedding below:

Prop 8 Ruling FINAL

Minority Report and Recent Advances in Pervasive Personalized Advertising

Several recent articles I've read about new developments in tracking and advertising in different countries - most of which reference the science fiction movie, Minority Report - reminded me of a quote often attributed to science fiction author, William Gibson:

The future is already here - it is just unevenly distributed

The articles describe the ways that various technologies - from special-purpose global positioning system (GPS) devices and face recognition software to general-purpose radio frequency identification (RFID) chips and web browsers - can be used to record information about us, and make it available to prospective advertisers in order to provide more contextually relevant advertising in a broader array of contexts.

The success of these increasingly pervasive personalized advertising systems depends, in part, on how they address three fundamental questions:

  • How much control do viewers of such advertising have over the information that is recorded?
  • What benefits do the viewers receive?
  • What risks do they perceive?

The Minority Report analogies refer to scenes in which iris scanning technology is used to identify shoppers in order to present customized messages. In the scene above, John Anderton (played by Tom Cruise) is offered audio and visual invitations to try products and services from Lexus ("the road you're on, John Anderton, is the road less traveled"), Guinness ("John Anderton! You could use a Guiness right about now.") and American Express Travel ("Get away, John Anderton; forget your troubles") as he makes his way through a mall on his way to board a train. People in this advertising scenario have little control over being identified (short of, say, eyeball transplants), do not appear to derive any direct benefit, and the risks of ubiquitous identification and tracking go well beyond potentially irritating personalized advertising: an elite Pre-Crime unit of the police may want to apprehend you before you commit a crime you don't even know you're going to commit. These factors may explain why reviews and reactions to this scenario, including the references in these recent articles, are almost universally negative.

OysterCard On Sunday, the Daily Telegraph published an article by Richard Gray on Minority Report-style advertising billboards to target consumers, describing a system being developed by the IBM Smarter Planet program in which RFID chips - using near field communication (NFC) with a range of about 4 inches (10 cm) - are used to identify people. Few additional details are provided about the system, but the technology suggests that some kind of explicit "check-in" will be required in order for people to be identified. The article alludes to the Oyster Cards used in the London Underground and other transportation systems as being a compatible technology, and the recent announcement that all of Nokia's new Symbian phones will come equipped with NFC in 2011 suggests that the availability of NFC-enabled devices will continue to grow. I can imagine a context in which "check-ins" used for one purpose (gaining access to a train platform) could be used for another purpose (targeted advertising on nearby displays and/or speakers). If customers are given the control to explicitly opt in to such a system, and were rewarded - perhaps by subsidized fares associated with their cards (or phones) - then I believe the benefits would be perceived as outweighing the privacy risks for at least some of the potential users.

NEC_NextGenerationDigitalSignageSolution Sunday's article references another Daily Telegraph article by Andrew Hough earlier this year with a similar theme - and a similar title ('Minority Report' digital billboard 'watches consumers shop') - but with a different technology. NEC is developing a Next Generation Digital Signage Solution that combines large displays, video cameras and face recognition software designed to determine the gender and approximate ages of the person or people in front of the display. An AFP report two weeks ago, Tokyo trials digital billboards that scan passers-by, refers to a Digital Signage Promotion Project in which 27 high-tech advertising displays were deployed in commuter stations in Tokyo. I suspect this is a pilot of the NEC system, although NEC is not mentioned anywhere in the report.

While potentially less invasive than the RFID-based approach - inferring age and gender rather than requiring individual identification - the use of cameras may instead be perceived as more invasive, depending on how people believe captured images are being handled, e.g., deleted, saved for internal use only or potentially sold to third parties. There is certainly less control afforded to potential users, aside from cloaking their faces as they pass by. The proposed benefits described in the earlier Daily Telegraph article appear to be targeted primarily toward the advertisers, who potentially will be able to target advertising toward specific demographic groups in proximity to the displays. However, the more recent Telegraph article suggests that personalized advertising might offer indirect benefits to viewers, in that they "may help reduce costs that are passed onto the consumer by reducing the amount of poorly targeted advertising" ... perhaps reflecting progress toward addressing a problem observed by John Wanamaker, the father of modern advertising:

Half the money I spend on advertising is wasted; the trouble is I don't know which half.

TrySomethingNewWithOmo Another advertising campaign - costing $1M - seeks to move pervasive personal advertising from public and semi-public places into the home. In an article asking Is Your Detergent Stalking You?, Laurel Wentz at Advertising Age reports that Omo Detergent has inserted GPS devices in 50 boxes of new, improved detergent scattered in stores throughout Brazil. Owners of the special boxes will be tracked down in their homes - or, I suppose, wherever the detergent box comes to rest for a period of time - whereupon they will be presented with a free video camera and invited to participate in a special company-sponsored event. I don't know enough about Brazilian culture to predict how consumers in that country will respond, but given that they are not informed whether or not any particular box of detergent can be tracked - it's a surprise - I can imagine reactions that may range from the kind of ecstatic joy expressed by those contacted by the Prize Patrol unit in Publishers Clearing House Sweepstakes commercials to the abject terror felt by those who are tracked down by John Anderton's Pre-Crime unit in Minority Report.

FacebookAdsMuffinTop The Wall Street Journal just launched a series of articles on Internet spyware that may be tracking information through your web browser. In the first installment, The Web's New Gold Mine: Your Secrets, Julia Angwin reports that the top 50 web sites installed 3,180 cookies, flash cookies or beacons on computers that visit their web sites, which are then used to either personalize ads shown to the user directly or sold to third parties who aggregate the data. Many users may already be aware of the use of web browser tracking technology, but the extent of the tracking may be disturbing to some, and considerable convenience must be forfeited in order to control the tracking. Perhaps even more disturbing is the potential for the algorithmically personalized advertising to intrude into intimate dimensions of our lives, resulting in psychological harm. The article describes a case of a 17 year old girl whose browsing behavior led to her being [correctly] categorized as someone interested in weight loss programs. Although accurate, the advertising was not always welcome: "I try not to think about it…. Then [the ads] make me start thinking about it." Two years ago, Rachel Beckman wrote a related article for the Washington Post on Facebook ads target you where it hurts, subtitled "My Facebook page called me fat", describing the recurring emotional pain experienced by a user repeatedly exposed to targeted advertising like the Muffin Top ad shown on the right.

The audio and visual advertisements soliciting John Anderton's attention were for products and services that he probably did not mind being associated with - luxury cars, beer and travel to exotic locales. Thinking back to issues I raised about the prospects of personalized, publicly displayed promotions of personal care products in a drug store (represented by OlayForYou screens at WalMart), I wonder what kind of reaction Anderton - and the movie audience - might have to personalized, publicly displayed advertisements for more intimate or personal products and services in a mall - say, Viagra, Hair Club for Men, or WeightWatchers (and as I type these, I can already envision some of the spam comments I'll get on this post).

CoCollage-Trabant Despite my concerns about potential abuses and/or unintended consequences that may arise in an era of increasingly pervasive personalized advertising, I believe that well-designed active environments that can sense and respond to people in contextually appropriate ways can offer benefits that outweigh the risks. My own work on proactive displays has involved the linkage of different sensing technologies - infrared badges, RFID, Bluetooth mobile phones and magetically-striped loyalty cards - to social media sites in order to bring some of the richness of what we share in our online social networks into the physical spaces we share with others. Revealing the interestingness of the people nearby, e.g., through showing their photos on a large display, creates new opportunities for enhanced awareness, appreciation, interactions and relationships. Although some of the users of these prototypes have found them intrusive or otherwise undesireable, many users found them sufficiently advantageous to explicitly opt in, and even those who have not opted in have enjoyed seeing the social media (mostly photos) shared by others on the nearby displays.

One of the challenges we have faced in these systems - which I sometimes describe as bridging the gaps between people by bridging the gaps between online and offline - is how to bridge the gap between cool research prototypes to sustainable and pervasive product or service. The only way I can envision this happening is with advertising revenue streams, which is the path we were pursuing with CoCollage. A few of the other current generation digital signage solutions include user-generated social media along with advertiser-generated media in their mix (e.g., LocaModa and Aerva), but I think there is still considerable room for next generation digital signage solutions to provide increased control and benefits for their users, to help compensate for real or perceived risks ... and avoid the currently inevitable comparisons to Minority Report.