Gumption

Thumbs down on Netflix ditching of 5-star ratings for a thumbs-based system

The Netflix video streaming and DVD service announced Thursday that it is switching from a 5-star rating system to a simpler thumbs up / thumbs down system. I've been a Netflix user (and fan) for many years, and love their personalized ratings predictions. I have often used their model in presentations and brainstorming involving other services that could benefit from that kind of personalization. I think this change is a bad idea.

I called the Netflix Help Center (866-579-7172), and the customer service representative I spoke with told me they were eager to receive feedback on this topic, especially feedback that specifies why users are in favor or not in favor of the proposed change. I shared several reasons why I thought it was a bad idea and I want to share those reasons here, in the hope it may encourage other Netflix users - especially those who share my view that it's a bad idea - to contact Netflix and provide feedback.

Granularity is Good

The main objection I have to the proposed change is that I make careful distinctions in both the ratings I give to movies I have seen and on the personalized predicted ratings Netflix offers for movies I have not yet seen. I probably watch, on average, 2 hours of TV a week, 1 DVD every month, 1 movie in a theater every 3 months. I hardly ever watch video content on Netflix, YouTube or other streaming sources. So I'm probably an outlier on several dimensions.

That said, on the rare occasions when I do want to watch a movie - at home or in a theater - I will only watch a movie for which the personalized predicted Netflix rating is at least 4.0. Since I know the personalized prediction accuracy is dependent (in part) on my own ratings, I am very careful in how I rate movies. I use the following interpretations for the 5-star scale:

• 5 stars: a movie I liked so much I've seen it several times and/or would enjoy seeing again
• 4 stars: a movie I liked a lot, but am not interested in seeing again
• 3 stars: a movie I liked, but would probably have preferred to spend my time watching something else
• 2 stars: a movie I didn't like, and probably didn't watch much of
• 1 star: I don't know if I've ever seen a 1-star movie, and certainly don't want to ever see one

While some people may find it easier to give a thumbs up or thumbs down rating (which I will refer to hereafter as a thumbs-based rating), I would find it more difficult. I envision the following mapping from my 5-star schema to thumbs-based ratings:

• Thumbs up for a 4-star or 5-star movie
• No rating for a 3-star movie
• Thumbs down for a 1-star or 2-star movie

Given that I rarely see a 2-star movie, I would probably only be giving thumbs up ratings in the proposed new scheme, and predict that the lower volume of ratings combined with the lower granularity of ratings would result in less accurate Netflix rating predictions.

Quality vs. Quantity

Speaking of quantity, the Verge article reported that Netflix saw a 200% increase in the number of ratings among the test group who used thumbs up or thumbs down, compared to the number of ratings using the 5-star rating group.

The article doesn't report on the change in the number of users who submit ratings using thumbs up or thumbs down, nor is it clear whether a specific control group was used in the experiment. Based on their marvelously detailed posts in the Netflix tech blog, especially the posts on their recommender systems, I suspect they were very careful in the way the designed the experiment. Perhaps more details will eventually be reported there.

The article also doesn't report on the quality of the recommendations under the thumbs-based rating system. More is not necessarily better, and it is not clear what kind of impact the increased quantity had on the perceived quality of predictions based on the new system.

Given that the average U.S. adult consumes 5.5 hours of TV, movies, games, and other video content per day, I suspect most users are less discriminating than I am with respect to what they will watch. It may be that the quality of recommendations using the new system serves high-volume - or even average-volume - video consumers as well or better than it would serve low-volume video consumers. But if my supposition that higher volume video consumers are less discriminating is correct, then the increase in quality may not have much impact on the amount consumed. And since Netflix charges flat monthly rates, those of us who consumer relatively little video content are paying just as much as those who consume large amounts of video content .. and if the recommendation quality declines for someone like me, who consumes little content, and the quantity of video I consume similarly declines, I am more likely to discontinue the service than a high-volume consumer who might consume less if the quality of recommendations is not as good (due to fewer ratings). But if they are already consuming a large quantity of video, I don't understand what problem is Netflix trying to address.

Returns on Investments

The article draws an analogy between Netflix ratings and Spotify thumbs-based ratings, which I think is an inappropriate comparison point. I use both the Spotify and Pandora streaming music services (in fact, I'm a paid subscriber for both (I hate commercials in any medium)), but rating a song that lasts a few minutes is very different - in my view - from rating a movie that lasts a few hours. I'm much more willing to provide a finer granularity rating (e.g., on a 5-star scale) for an experience that will last hours vs. minutes.

I think a better comparison point would be Yelp, which uses 5-star ratings for restaurants and other service providers. I'm willing to provide ratings on a 5-star scale for restaurants, because it represents a more significant investment of time (and money). I would even consider TripAdvisor, an online service for reviews and ratings of hotels and other destinations and activities associated with traveling, a better comparison point than Spotify, as pl

Personalized Ratings for All

In fact, I think both Yelp and TripAdvisor could benefit from adopting the potentially-soon-to-be-former Netflix personalized rating scheme. I am growing weary of wading through reviews of restaurants on Yelp from people who rant about the bartender not paying attention to them, or a special event dinner that went awry, or from anyone who doesn't share similar tastes in restaurants to me. I would love it if Yelp would offer a personalized rating, or at least let me read reviews from people like me.

TripAdvisor ratings have become almost useless to me. It appears that many hotels are carpet-bombing guests with email invitations to review their stay, and the result seems to be that many places now have an overwhelming abundance of reviews from people who have only posted one review. I consider most newbie reviews nearly useless, both because they tend to be short and uninformative, and because there is no way to know what kind of other places the person has reviewed, so I can't tell how much the reviewer is like me.

I could rant further on the decline of both of these services - which I once found far more useful - but I will let it go (for now). I wanted to compose this post because throughout all the years I've been a Netflix user, the service has only gotten better (as I gave it more ratings upon which to make recommendations), and I'd hate to see yet another beloved rating, review and recommender service decline.

If you feel similarly, I urge you to call Netflix soon, as they are reportedly planning to roll out the new thumbs-based rating system in April.

Spiritual reparenting: severed belonging, benefactors, vulnerability & reconnection

Tara Brach's weekly dharma talks and guided meditations have been a consistent source of inspiration and equanimity for me over the past several years. A few weeks ago, she presented a talk on spiritual reparenting to which I've listened four times (thus far), as it resonated on many levels for me. The practice of becoming one's own loving parent is a recurring theme for adult children of alcoholics and other dysfunctional families, and I find Tara's perspective on the topic offers a valuable contribution to my understanding of the hurting and healing processes. I was so moved by the wisdom presented in the talk that I transcribed the entire 52 minute episode. I'll embed the podcast below, and include a link to the unofficial (and unauthorized) transcription, but I also want to share a few highlights here.

One of the themes Tara emphasizes is the wound of severed belonging, which often occurs while we are children, when we get the explicit or implicit message that we are not okay - not good enough - from parents or other authority figures. In the resulting trance of unworthiness, we feel the pain of separation - of being disconnected - from those we love .. and from whom we want to feel love. Separation anxiety can afflict us at any age, in the context of any significant relationship, though we are especially vulnerable when we are young and inexperienced. As I've noted in several of my most recent blog posts - the pain of severed belonging can continue to affect us for many years after the infliction of the wound.

Fortunately, many of us encounter a benefactor, someone who offers us unconditional love and a healing presence at a time when we are in deep pain. A benefactor repairs the severed belonging and helps us feel connected again: loved, and thereby, worthy of love. I am grateful for the appearance of benefactors at different stages of my life, when I was suffering with the pain of disconnection and the trance of unworthiness.

in her talk, Tara asks "What are the qualities in that person that made a difference to you?", and in reflecting on my benefactors, I see that the common qualities were honesty, self-awareness, compassion and vulnerability. Vulnerability is especially relevant to the process of repair and reconnection. Any benefactor who has appeared in my life has been willing and able to relate to my suffering by revealing one or more elements from their own experience that mirrors or closely aligns with my wound and acknowledging that the wound hurts.

I often magnify my suffering through self-inflecting a second wound - what Tara calls a "second arrow" - by judging myself defective and weak, believing that the wound shouldn't hurt so much, that there's something wrong with me. Simply knowing that I have a fellow traveler, especially one that I admire and respect, helps reduce my sense of isolation and restore a feeling of belonging and worthiness.

I count Tara among my benefactors, even though I've never met her. In her books and talks, she regularly shares stories from her own experience that exhibit the kind of vulnerability that creates an opening for connection and reconnection. Her practice of RAIN (Recognizing, Allowing, Investigating and Nurturing) has been an important component in my journey of letting go of emotions. I highly recommend her talks, as the calm, soothing voice through which she expresses her honesty, self-awareness and compassion is an especially welcome departure from the tone of much of the political discourse - and discord - that is arising from the lack of honesty, self-awareness, compassion and vulnerability exhibited by some who have recently come to power in the United States.

I also recommend adopting the practice of financial beneficence (or dana), and joining me in becoming a contributor in supporting Tara's talks and meditations, which she offers for free. I believe dana will be an increasingly important way to support the voices I want to hear, as more voices I don't want to hear are given more prominent platforms, supported at taxpayer expense.

I'll include the abstract for Spiritual Parenting below, since the highlights above only focus on a small portion of the talk. The talk includes references to other inspiriting sources of wisdom - such as a quote by attachment science researcher Louis Cozolino on the survival of the nurtured, Krista Tippett's On Being interview with Ruby Sales ("Where does it hurt?"), Martin Luther King's speech on "Beyond Vietnam: A Time to Break the Silence", and the 1998 movie, The Horse Whisperer.

When we are not sufficiently nurtured in childhood, we are inclined toward anxiety, depression, addiction and other forms of suffering. In a deep way, we do not feel at home with others. We are disconnected from our own body, heart and spirit.

This talk explores how meditation offers “spiritual reparenting” as we learn to bring interest, understanding and love to our own inner vulnerability. This process of healing extends to our relationships with others and our larger society – by reaching out to widening circles with interest and care, we bring increasing harmony and peace to our world.

From a place of caring, ask, “Where does it hurt?”

The podcast is embedded below for easy access. Here's a link to my unofficial transcription of Spiritual Reparenting. Namaste.

Letting Go of Emotions vs. Thoughts

A while back, I wrote about letting go of blame and judgment, following an enlightening "Zen with Len" retreat I attended last January. While the retreat helped me release others from anger I was feeling about perceived betrayals or other wrongs, I was still left with my feelings of pain, sadness and fear in response to those experiences. Letting go of pain has been an evolving process for me, and I wanted to share a few resources and practices that I have found useful on this part of the journey.

David Hawkins literally wrote the book on "Letting Go" (subtitle: "The Pathway of Surrender"), in which he argues for the primacy of emotions over thoughts, and proposes that by letting go of emotions, we can let go of the thoughts that emanate from those emotions.

Thoughts are merely rationalizations of the mind trying to explain the presence of the feeling... The thoughts associated with even one feeling may literally run into the thousands. The understanding of the underlying emotion and its correct handling is, therefore, more rewarding and less time-consuming than dealing with one's thoughts.

I tend to focus most of my time, energy and attention on thinking rather than feeling, and have always believed that thoughts give rise to feelings, but Hawkins suggests it is the other way around:

The mind is .. a survival mechanism, and its method of survival is primarily the use of emotions. Thoughts are engendered by the emotions and, eventually, emotions become shorthand for thoughts .. Reason is the tool the mind uses to achieve its emotional ends.

I initially found this description and the prescription for the "correct handling" of emotions counter-intuitive and unsettling. In my efforts to let go of the lingering pain, I had been devoting some of my meditations to focusing on troubling - and sometimes traumatic - thoughts, and sitting with the thoughts in an effort to let go of the emotions they give rise to. Hawkins suggests switching that around, and focusing directly on the emotion:

Letting go involves being aware of a feeling, letting it come up, staying with it, and letting it run its course without wanting to make it different or do anything about it. It means simply to let the feeling be there and to focus on letting out the energy behind it .. It is resistance that keeps the feeling going .. A feeling that is not resisted will disappear as the energy behind it dissipates... The feeling can .. be worked with by first accepting that it is there, without resisting it or condemning it. And then one begins to empty out the energy of the feeling directly by letting it be what it is until it runs out.

He acknowledges that some emotions are simply too overwhelming, and may require multiple letting go "sessions" of sitting with the emotions to release their power. It may also be helpful to deeply traumatic emotion into subcomponents, and let go of different elements incrementally over time.

Hawkins observes that suppressing emotions can increase stress, and that increased stress can lead to disease, and describes various ways that the mind and body are connected, much of which is consistent with my own experience, and with other teachings I've encountered (e.g., Gabor Mate's book, "When the Body Says 'No'"). He also covers several themes I wrote about in my last post, about attachment, dependency and possessiveness in relationships, so I won't go further down that path here, except to note here that I found one of his predictions about the potential outcomes of attachment especially poignant:

The other person, now feeling pressured by our energy of dependency and possessiveness, has an inner impulse to run for freedom, to withdraw, to detach and do the very thing that we fear the most.

One of the themes in the book I find most difficult to accept is the prescription to let go of desire, which echoes ideas I've encountered in [other] new age teachings:

wanting blocks receiving it [what is wanted] and results in a fear of not getting it. The energy of desire is, in essence, a denial that what we want is ours for the asking.. We surrender the emotion of desire and, instead, merely choose the goal, picture it lovingly, and allow it to happen because we see that is already ours.

Perhaps I'm not yet sufficiently evolved, but I personally don't see how wanting and desire - which are to be let go - are different from asking for something or setting goals - which are to be embraced - so I've decided to let go of this aspect of the teachings, at least for now.

There are a number of other aspects of the book that I find more difficult to accept. Hawkins was a proponent of using applied kinesiology to test muscle strength in the arm while a subject repeats a statement to "objectively" assess a subject's level of consciousness. His enumeration, ordering and descriptions of these levels - shame, guilt, apathy, grief, fear, desire, anger, pride, courage, neutrality, willingness, acceptance, reason, love, joy and peace - make intuitive sense to me, but associating frequency measurements with them, and estimating what proportion of the population has achieved these levels, do not. An essay on The Emperor's New Clothes: David Hawkins' Absolute Calibration of Truth offers a deeper investigation into some of Hawkins' questionable claims, and some of the ways he and his associates have actively sought to eliminate criticism from Wikipedia and other web sites.

As unsettling as I find some of these aspects of his teaching and behavior, I often find it useful to apply the slogan "take what you like and leave the rest", and continue to find resonance in his ideas about the primacy of emotions and the practices of letting go of the negative emotions ... although I will admit I initially found these unsettling as well.

One of the initially disturbing - but eventually resonant - aspects of Hawkins' views is that disturbances are beneficial:

Every life crisis carries within it the kernels of a reversal, a renewal, an expansion, a leap in consciousness, and a letting go of the old and a birth of the new .. a hidden lesson... Carl Jung [concluded that] there is an inborn drive in the unconscious toward wholeness, completeness, and realization of the Self, and that the unconscious will devise ways and means of bringing this about even if they are traumatic to the conscious mind...

One benefit of a crisis is that it often brings us into familiarity with our shadow .. Once the shadow has been acknowledged, it loses its power... Passing through a life crisis, then, makes us more human, more compassionate, more accepting and understanding of ourselves and others. We no longer have to indulge in making others wrong or making ourselves wrong.

... One major loss can awaken us to the nature of all attachments, and all relationships.

Indeed, I am hoping this journey through unresolved grief I started over a year ago is serving to awaken me to a higher state of enlightenment. The theme of crises and breakdowns facilitating growth has played a significant role in this awakening, and is a concept into which I plan to delve more deeply in a future post.

For now, I'll conclude with two other sources of inspiration for letting go that I revisit regularly. One is a brief passage by Tilopa shared in a guided meditation by Tara Brach - who regularly encourages attending to feelings in the body and letting go of thoughts in the mind - on Relaxing Open.

Let go of what has passed.
Let go of what may come.
Let go of what is happening now.
Don’t try to figure anything out.
Don’t try to make anything happen.
Relax, right now, and rest.

The other is a poem by Danna Faulds, from her book, Go In and In: Poems from the Heart of Yoga, which I read nearly every day.

Let go of the ways you thought life
would unfold; the holding of plans
or dreams or expectations – Let it
all go. Save your strength to swim
with the tide. The choice to fight
what is here before you now will
only result in struggle, fear, and
desperate attempts to flee from the
very energy you long for. Let go.
Let it all go and flow with the grace
that washes through your days whether
you receive it gently or with all your
quills raised to defend against invaders.
Take this on faith; the mind may never
find the explanations that it seeks, but
you will move forward nonetheless.
Let go, and the wave’s crest will carry
you to unknown shores, beyond your
wildest dreams or destinations. Let it
all go and find the place of rest and
peace, and certain transformation.

I continue to find it hard to accept that "the mind may never find the explanations that it seeks", but perhaps I'm moving forward, nonetheless.

Marriage, Romantic Realism and Relationship without Attachment

I've been a hopeless romantic for most of my life, but I am becoming increasingly disillusioned about love. I don't see disillusionment as a negative thing, but as a positive process: letting go of illusions, and thus becoming less romantic - and more realistic - about relationships ... especially long-term committed relationships ... especially the one I am in. I don't love my wife any less, but I'm striving to love her differently, more maturely, with greater acceptance and less expectation or attachment.

This maturation / disillusionment has evolved over many years, often unfolding in fits and starts. I may never complete the process, but I'm making progress. I wanted to share a few notes on the journey here, to collect and synthesize some relevant influences I've encountered along the path, both for my own future reference and in case it may be helpful to others.

One of the first realistic descriptions of a healthy relationship I encountered was in Anne Morrow Lindbergh's timeless tome, Gift from the Sea. My wife and I included a passage from this inspiring book in the readings at our wedding.

A good relationship has a pattern like a dance and is built on some of the same rules. The partners do not need to hold on tightly, because they move confidently in the same pattern, intricate but gay and swift and free, like a country dance of Mozart's. To touch heavily would be to arrest the pattern and freeze the movement, to check the endlessly changing beauty of its unfolding. There is no place here for the possessive clutch, the clinging arm, the heavy hand, only the barest touch in passing. Now arm in arm, now face to face, now back to back -- it does not matter which because they know they are partners moving to the same rhythm, creating a pattern together, and being invisibly nourished by it.

The joy of such a pattern is...the joy of living in the moment. Lightness of touch and living in the moment are intertwined. One cannot dance well unless one is completely in time with the music, not leaning back to the last step or pressing forward to the next one, but poised directly on the present step as it comes... But how does one learn this technique of the dance? Why is it so difficult? What makes us hesitate and stumble? It is fear, I think, that makes one cling nostalgically to the last moment or clutch greedily toward the next. [And fear] can only be exorcised by its opposite: love.

My wife has always been the more pragmatic and realistic partner in our relationship, probably due both to her general temperament as well as her greater experience with prior relationships before the two of us got together. I have tended to operate more toward the idealistic end of the spectrum, and have at times practiced more of a possessive clutch than a bare touch in passing. Fortunately - although it hasn't always seemed that way - the wisdom embodied in these two paragraphs has been re-presented through many lessons in many contexts over the years.

A related source of wisdom is David Schnarch's book, Passionate Marriage: Keeping Love & Intimacy Alive in Committed Relationships, which I first encountered - and found very unsettling - several years after we were married. Schnarch describes marriage as a crucible in which the role of our partner is not so much to be affectionate, appreciative and approving - though some partners may exhibit any or all of these characteristics at times - but to support us in our growth as differentiated and resilient individuals.

In my last post, I wrote about the pitfalls of depending on external validation. Schnarch contrasts external validation - or what he calls other-validated intimacy - with self-validated intimacy, in which "you don't expect your partner to validate or accept what you disclose. You validate yourself by showing your partner who you really are." I'll include a few other resonant passages from the book below.

We're driven by something that makes us look like we crave intimacy, but in fact we're after something else: we want someone else to make us feel acceptable and worthwhile. We've assigned the label "intimacy" to what we want (validation and reciprocal disclosure) and developed pop psychologies that give it to us - while keeping true intimacy away. We've distorted what intimacy is, how it feels, how much we really want it, and how best to get it. Once we realize that intimacy is not always soothing and often makes us feel insecure, it is clear why we back away from it.

...

Our self-made crises are custom-tailored, painstakingly crafted, and always fit perfectly. We construct emotional knots until, eventually, we are willing to go through them.

...

Differentiation involves balancing two basic life forces: the drive for individuality and the drive for togetherness. Individuality propels us to follow our own directives, to be on our own, to create a unique identity. Togetherness pushes us to follow the directives of others, to be part of a group.

Lack of differentiation alienates us from those we love. Emotional fusion deceives us into thinking that we're not connected and we move away in defense. But the deeper truth is that we have to move away to counterbalance the tremendous impact we feel our spouse has on us.

A more recent source of wisdom describing a healthy, illusion-free, non-attached way of relating to one's partner that I encountered is from Alain de Botton's evolving online compendium, The Book of Life. One chapter presents Romantic Realism: a term used to describe "a correct awareness of what can legitimately be expected of love". A reformulation and refinement of some of the wisdom in that chapter (and some of his other writings) appeared in another chapter, On Marrying the Wrong Person, a version of which became the most read article of 2016 in the New York Times. To my way of thinking, despite its title, this chapter/essay is not so much about marrying the wrong person as it is about marrying any person based on the wrong goals or expectations. Here are a few of the passages I find most resonant:

All of us are crazy in very particular ways. We’re distinctively neurotic, unbalanced and immature, but don’t know quite the details because no one ever encourages us too hard to find them out. An urgent, primary task of any lover is therefore to get a handle on the specific ways in which they are mad. They have to get up to speed on their individual neuroses. They have to grasp where these have come from, what they make them do – and most importantly, what sort of people either provoke or assuage them. A good partnership is not so much one between two healthy people (there aren’t many of these on the planet), it’s one between two demented people who have had the skill or luck to find a non-threatening conscious accommodation between their relative insanities.

...

The problem is that knowledge of our own neuroses is not at all easy to come by. It can take years and situations we have had no experience of. Prior to marriage, we’re rarely involved in dynamics that properly hold up a mirror to our disturbances. Whenever more casual relationships threaten to reveal the ‘difficult’ side of our natures, we tend to blame the partner – and call it a day. As for our friends, they predictably don’t care enough about us to have any motive to probe our real selves. They only want a nice evening out. Therefore, we end up blind to the awkward sides of our natures. On our own, when we’re furious, we don’t shout, as there’s no one there to listen – and therefore we overlook the true, worrying strength of our capacity for fury. Or we work all the time without grasping, because there’s no one calling us to come for dinner, how we manically use work to gain a sense of control over life – and how we might cause hell if anyone tried to stop us. At night, all we’re aware of is how sweet it would be to cuddle with someone, but we have no opportunity to face up to the intimacy-avoiding side of us that would start to make us cold and strange if ever it felt we were too deeply committed to someone. One of the greatest privileges of being on one’s own is the flattering illusion that one is, in truth, really quite an easy person to live with.

...

We believe we seek happiness in love, but it’s not quite as simple. What at times it seems we actually seek is familiarity – which may well complicate any plans we might have for happiness.

We recreate in adult relationships some of the feelings we knew in childhood. It was as children that we first came to know and understand what love meant. But unfortunately, the lessons we picked up may not have been straightforward. The love we knew as children may have come entwined with other, less pleasant dynamics: being controlled, feeling humiliated, being abandoned, never communicating, in short: suffering.

One of the best definitions of suffering I've encountered is from Tara Brach, who has shared the following formula from the Four Noble Truths in a few of her dharma talks:

suffering = pain X resistance

I think the risk of pain always accompanies the prospect of intimacy in a relationship: anyone we care enough about can say or do things - or not say or do things - that hurt us. One may not be able to eliminate pain in a relationship, but one can reduce resistance by acceptance of what is ... and letting go of attachment to what is not.

And the themes of abandonment, suffering, acceptance and letting go of attachment bring me full circle to yet another gem of wisdom I gleaned from a recent re-reading of Anne Morrow Lindbergh's Gift From The Sea, with which I'll close:

When you love someone, you do not love them all the time, in exactly the same way, from moment to moment. It is an impossibility. It is even a lie to pretend to. And yet this is exactly what most of us demand. We have so little faith in the ebb and flow of life, of love, of relationships. We leap at the flow of the tide and resist in terror its ebb. We are afraid it will never return. We insist on permanency, on duration, on continuity; when the only continuity possible, in life as in love, is in growth, in fluidity - in freedom, in the sense that the dancers are free, barely touching as they pass, but partners in the same pattern.

The only real security is not in owning or possessing, not in demanding or expecting, not in hoping, even. Security in a relationship lies neither in looking back to what was in nostalgia, nor forward to what it might be in dread or anticipation, but living in the present relationship and accepting it as it is now. Relationships must be like islands, one must accept them for what they are here and now, within their limits - islands, surrounded and interrupted by the sea, and continually visited and abandoned by the tides.

External Validation and Emotional Responsibility

Years ago, I discovered Validation, a wonderful short film by Kurt Kuenne about a parking garage attendant who validates tickets ... and validates the people who bring their tickets to the window, offering compliments to each person as he stamps their ticket "Validated". He goes on to offer validation far and wide beyond the parking garage, and every recipient of his validation is uplifted by the experience, until he encounters one person who seems unaffected by his compliments and other efforts to make her happy.

As with many sources of inspiration, each time I revisit the source, I notice new aspects, since I am a different person during each visit. I watched the film again recently, after returning from a meditation retreat, where my main takeaways were letting go of blame and judgment. One of the other lessons from the retreat, noted near the end of my post, was

• Teaching me that one cannot depend on anyone else for validation

During this most recent re-watching of Validation, I was feeling uncomfortable and irritated during the first portion, where people are literally lining up to bask in the glow of external validation provided by the parking lot attendant. I realized I was judging them for their dependency, and then recognized that the dependency on external validation that irritated me in others was the same dependency that irritated me about myself ... as so often happens, my externally-focused Judge was simply reflecting my internal Critic.

The aspect of the film that resonated especially deeply with me this time was the unsuccessful efforts of the parking lot attendant to validate - and uplift - a woman who worked as a photographer at a motor vehicles department. Nothing he tried seemed to have any effect on the woman's disposition, and I realized that his own sense of validation and worthiness was dependent on other people's feeling validated in response to him. He was happy when he made other people happy ... and he became unhappy when he could not make this one person happy ... i.e., he was suffering from dependence on external validation.

This reminded me of another lesson I learned at the retreat:

• No one can make me angry, shame me or cause me to dissociate

Watching the movie, I realized two important corollaries to this:

• No one can make me happy
• I cannot make anyone else happy (or angry or sad ... or cause any other emotion)

I might - and often try to - create conditions that I believe are more conducive to someone else feeling happy, but I cannot make them happy. Conversely, I might - and often try to - avoid creating conditions that may be conducive to someone else feeling angry, but I cannot make them angry. So the expanded lesson is:

I am not responsible for anyone else's emotions, I am only responsible for my own.

This may be obvious to many emotionally mature individuals, but as a life-long people pleaser with an inner conviction of intrinsic unworthiness, I have relied on others' validation for as long as I can remember. I believed others were responsible for my validation and I was responsible for theirs

I have recognized repeated opportunities to learn this lesson since returning from the retreat. I'm sure such opportunities have always abounded, I'm just now becoming more aware of them (when the student is ready, the lesson is recognized). When I do something with the goal of winning another's expression of appreciation or approval - or avoiding another's expression of disapproval - I am setting myself up for disappointment, and invalidation.

Since the retreat, I have new tools to look at that disappointment, take responsibility for the feelings of sadness and anger, and recognize that the feelings are a signal that I was doing something for the wrong reasons: seeking approval vs. doing something because it is the right, or kind, thing to do. And in this recognition, I can shift from feeling blame and judgment to feeling grateful for yet another unsought opportunity for further awakening

Related articles

There are no mistakes only lessons; a lesson is repeated until learned; tuition varies

Letting Go of Blame and Judgment: Emotional Transformation through Zen with Len

Many years ago, I experienced betrayal, pain and anger about something that someone did. I did not have the tools or life experience to fully understand my reactions - much less transform them - at the time, and the memory of the episode has resurfaced periodically, re-triggering unresolved emotions. The most recent recurrence began last October, disrupting my ability to sleep and negatively impacting other dimensions of my life over several months. The spell was finally broken during a retreat in January that was organized and led by my good friend, Kensho Len Silverston, promising - and delivering - emotional transformation. I wanted to share some of the insights and experiences that contributed to my breakthrough(s) that weekend.

I went into the retreat filled with blame and judgment: blaming the other person for causing my pain and anger, and judging that what the other person did was wrong. The intensity of my righteous indignation was magnified by the person's unwillingness to admit any wrongdoing or express any regret about the episode. I knew it was up to me to resolve whatever lingering emotions I was holding on to - as Tara Brach so aptly puts it, the boundary to what we can accept is the boundary to our freedom - and it was increasingly clear, after several months of disequilibrium, that I needed help. When the student was ready, the retreat appeared.

Meditation, Yoga and Qigong

The Zen with Len retreat consisted of several days of meditation, gentle yoga and Qigong sessions - all interleaved so as to balance our focus on mind and body - as well as special sessions exploring a number of other practices Len has found helpful in his own emotional transformations. Most meditation sessions were a half hour long - a bit of a stretch for me, as I had previously only sat for a maximum of 20 minutes at a time, but not out of my comfort zone - and I found the yoga sessions were similarly within my comfort zone, offering opportunities to gently push the edges.

While I have practiced meditation and yoga (with varying degrees of dedication) before, the Qigong exercises - a type of moving meditation - were new to me. One exercise, in particular, seemed like it was custom designed for what I was seeking at the retreat, and proved to be one of the most significant sources of breakthrough: an invitation to bring into consciousness something we want to let go, and then manifesting the letting go through movement (not just words). I'll embed a 15-minute Qigong video Len made after the retreat below; the letting go exercise can be found around the 9:55 mark.

The Qigong exercises were repeated several times throughout the weekend, and for each iteration of the letting go exercise, I alternated between letting go of blaming the other person, and letting go of my judgment of what the other person did. By the end of the weekend, I succeeded in letting go of both. However, I soon found that the blame and judgment were simply redirected toward me: how could I have tormented myself all these years? Fortunately, there were other tools provided during the weekend to help with this.

Mondo Zen

One of the special afternoon sessions was on Mondo Zen, a protocol adapted from ancient Zen principles by Len's teacher, Jun Po, to facilitate the awakening of Clear Deep Heart / Mind through a process of koans. The process - detailed in the Mondo Zen training manual - is divided into two parts. The first part, Ego Deconstruction/Reconstruction Koans, is designed to help loosen one's grip on traditional ways of seeing, understanding and acting and arrive at a state of "not knowing". I had worked through similar processes in the Warrior Monk retreat I attended several years ago, and found the refresher helpful. However it was the second part, Emotional Awareness Intervention Koans, that really set the stage for the transformation I experienced.

The resurgence of the episode that brought me to the retreat triggered a number of different reactions at different times: anger, pain, shame and dissociation. In Mondo Zen, I learned that

• No one can make me angry, shame me or cause me to dissociate / disconnect
• Any anger, shame and/or disconnection I feel is a reaction to fear and/or sadness
• Fear and/or grief is rooted in deep caring

I also learned that all of these emotions involve some kind of violence:

• Anger is violence against others
• Shame is violence against myself
• Disconnection is violence against a relationship

Through the 2-hour Mondo Zen exercise during the retreat - a highly abbreviated, but effective, version of what is typically a multi-day retreat of its own - I was able to

• Understand and accept that I reacted with anger, shame and disconnection to the past episode
• Take full responsibility for my reactions
• Recognize and take responsibility for the harm I have caused myself and others through my anger, shame and disconnection
• Accept that the other person did nothing wrong

All of these new insights reinforced my ability - and willingness - to let go of blame and judgment.

Voice Dialogue

Another practice Len introduced in a special session was Voice Dialogue. We all have a multitude of voices in our heads, each representing different selves or parts of our personality. Each voice serves us in some positive way, and each voice has a different level of prominence in each of us, both in general and in the context of any particular inner conversation. Each voice can be harmful if allowed to commandeer the conversation to the exclusion of other voices. Disowning or rejecting any voice can also be harmful, as the abandoned voice will always find some way to leak out and express itself. We also have a higher self, or "True Nature", that serves as a moderator of our inner dialogues. I like to think of the voices as an inner program committee or board of directors, and my "True Nature" as the chairman of the board.

As I understand it, there are a few different variations on the number and specific labelings of voices. The one Len used was from the book - and associated deck of 52 cards (each representing a different voice) - Selves in a Box. I think the most important aspect of the practice of Voice Dialogue is not so much the specific labels that are used for different voices, but the act of explicitly labeling the voices itself, and the way this differentiation enables one to identify and consciously moderate among the voices ... a manifestation of the principle I have heard articulated by several different spiritual teachers (including Tara Brach and Dan Seigel):"If you can name it, you can tame it".

Among the most prominent voices on my board of directors (in alphabetical order) are

• The Accommodator
• The Critic
• The Judge
• The Loner
• The Perfectionist
• The Romantic
• The Sensitive
• The Thinker
• The Vulnerable Child

I have very loud and strident Critic (inwardly directed) and Judge (outwardly directed) voices. As I mentioned above, when I let go of blame and judgment of the other person, I redirected the blame and judgment toward myself, unconsciously shifting the leading voice of that inner conversation from the Judge to the Critic. Recognizing the rise of the Critic enabled me (or my True Nature) to explicitly call upon The Nurturer to comfort The Vulnerable Child so that I could better practice self-acceptance and self-forgiveness ... a challenging practice that is still unfolding for me.

The Demartini Breakthrough Experience

The final special session that Len led us through was an abbreviated version of the Demartini "Breakthrough" Experience, which is another process that is typically offered in a multi-day retreat of its own.

The Breakthrough process is based on the recognition that all traits have costs and benefits, and so traits I judge as "bad" also offer benefits to me and others. The process involves a sometimes painful investigation into what those hidden benefits might be. During the retreat, I chose to work on the trait of remorselessness in the person I had formerly blamed for my anger and pain, a trait I had previously labeled as uniformly bad.

Through an iterative process of excavation through layers of resistance, I came to recognize that this trait provided me a number of benefits, including:

• Teaching me how to be unapologetically true to oneself (vs. consumed with people pleasing or accommodation of others)
• Teaching me that it is OK to do what one wants, without undue regard for how someone else might feel about it, if it does not violate an agreement or directly harm another person
• Teaching me that one cannot depend on anyone else for validation
• Offering me an unsought opportunity to accept full responsibility for my reactions, which I can apply to other situations in which I experience betrayal, anger and/or pain

This last point represents a significant and unexpected breakthrough, helping to reinforce some of the other dimensions of transformation I experienced during the retreat. If the other person had expressed remorse, it would have vindicated my feelings of blame and judgment, enabling me to avoid looking any more deeply into my emotional reactions, and thereby avoid taking responsibility for them. The lessons to be true to myself, not depend on others for validation, and take full responsibility for my reaction are lessons that have been repeated at various times, in various ways and at various costs over many years.

I believe the prolonged intensity of pain that preceded the latest course offering was necessary for the breakdown that facilitated the breakthrough, and I am grateful for the timely, multi-dimensional learnings offered at the Zen with Len retreat.

"Expressed Emotions" in Everyday Interactions: Acceptance vs. Intervention with Family and Friends

I've listened to the most recent episode of NPR's Invisibilia, The Problem with the Solution, three times in three days, crying a little less - and understanding a little more - each time I listen. I believe the emotional impact stems from my experience as a son, a husband, a father and a friend ... and more specifically, how "expressed emotions" - criticism, hostility and emotional over-involvement (essentially, expressions of non-acceptance) - have affected me and those I love.

The Invisibilia story focuses on mental illness, and how the traditional American mindset of wanting to fix problems can be counterproductive in dealing with people who have mental illness, i.e., the problem is [our preoccupation with] the solution. In the town of Geel, Belgium, people with mental illness are boarded by townspeople for, on average, 28 years. Studies have shown that people with mental illness who are placed in community settings (like Geel) after treatment are less likely to relapse than when they return home to live with their own families. This is likely due - in part - to the fact that host families are not related to their boarders, and are thus not attached to or over-involved in fixing or curing their boarders' mental illness. Instead, the boarders are accepted as they are.

As a resident of the Broadway Housing Communities - a project for recreating the culture of Geel in America - so aptly noted in an Invisibilia interiew, "Everybody has a touch of mental illness". Everyone I know has at least gone through periods where they have experienced "an unhealthy condition of body or mind", and many of those I know best have suffered a great deal from chronic unhealthy thoughts and emotions. In fact, it is the revelation of sufferings - and the sharing of how we are working through some of these unhealthy thoughts and emotions - that have formed the basis of the strong bonds I feel with my closest friends. And I have a growing appreciation for the suffering experienced by people I don't know well, and the way that suffering underlies some of the behaviors I find hardest to accept. As William Wordsworth observed:

If we could read the secret history of our enemies, we should find in each man's life sorrow and suffering enough to disarm all hostility.

I don't mean to equate the suffering of what some call "the worried well" with the suffering of those with serious schizophrenia, bipolar disorder or other types of mental illness highlighted in the Invisibilia story. I just want to say I can relate to suffering emanating from chronic unhealthy thoughts and emotions ... and to the effect of acceptance vs. intervention on that suffering.

I believe a great deal of suffering is caused by emotional over-involvement, and the various verbal and non-verbal ways that family members - especially parents (or grandparents) - try to "improve" us.

Invisibilia co-host Lulu Miller, who included her oldest sister's mental illness - and her family's response to it - as a recurrent theme throughout the story, noted that her older sister "had been quietly accepting this message that she was somehow not good enough" throughout her childhood, and the part that breaks me down and gets me sobbing every time (even now, simply copying and pasting the following text from the show transcript), is her father's candid admission of insufficient empathy in dealing with his oldeest daughter:

She was 2 or 3. And she was having a huge freak out at 3 in the morning, just screaming and yelling in her bed. And I went in there, and I got angry at her. And I sat on her bed, and I was trying to calm her down. And I was sort of holding her shoulders, and I got angry at her. I said stop it, stop it. And I remember in her tears - as I was getting angrier, in her tears, she kept saying I can't help it. I can't help it. And I didn't hear that. It's - it's something I will always regret that my feeling anger instead of empathy - didn't know what I was doing as a father.

This feeling of insufficiency resonates deeply with me, and not just in my role as a father. If I had to single out the unhealthy thought that has created the most emotional suffering throughout my life, it is that I am not good enough. I have written before about my theory of how this inner conviction of unworthiness evolved; here, I'll simply note that I trace part of the origins partly to the effects of growing up in a household in which a family member suffered from mental illness - my father's alcoholism - and partly to the effects of my maternal grandfather's efforts to "improve" me through what I now recognize as "expressed emotions".

As an adult, I have found that some of my wife's well-intentioned efforts to "improve" me have inadvertently only served to deepen the void, and I am sure some of my efforts to "improve" her have had similar unintended consequences, especially when expressed with anger rather than empathy. I know both of our children have suffered from some of their parents' efforts to "improve" them. My son has confirmed that the two times that I exploded in anger toward him - rather than opening to empathy for his underlying suffering that gave rise to the behaviors to which I was reacting - both had a significant negative impact on him, and at least one of my daughter's lingering emotional wounds was inflicted by an angry expression of parental disapproval intended to "improve" her.

Which leads me to wonder whether acceptance always trumps intervention when interacting with family members. We naturally want our spouses and children to be the best they can be, and part of the role of a parent is to teach our children how to thrive. That said, I believe that "expressed emotions" (as defined above) are always harmful to both the senders and receivers of the expressions. I suppose that there are ways to encourage "improvement" that do not involve criticism, hostility or emotional over-involvement - the improv comedy practice of using "yes, and ..." rather than "yes, but ..." come to mind - but and in interpersonal interactions, I find it challenging to seek or promote improvement in another person without at least implying that someone is not good enough. Amid my increasing uncertainty, I am more and more inclined to err in the direction of acceptance rather than intervention.

In my journey toward greater acceptance, I have encountered a number of other sources of inspiration that are well aligned with the wisdom expressed in the Invisibilia episode. Before closing, I want to share a few of these here.

Tara Brach teaches that one of the ways that we perpetuate the trance of unworthiness is by focusing on the belief that something is wrong, something is missing, and the way to break out of this trance is to embrace a radical acceptance of ourselves and others.

Both our upbringing and our culture provide the immediate breeding ground for this contemporary epidemic of feeling deficient and unworthy. Many of us have grown up with parents who gave us messages about where we fell short and how we should be different from the way we are. We were told to be special, to look a certain way, to act a certain way, to work harder, to win, to succeed, to make a difference, and not to be too demanding, shy or loud. An indirect but insidious message for many has been, “Don’t be needy.” Because our culture so values independence, self-reliance and strength, even the word needy evokes shame. To be considered as needy is utterly demeaning, contemptible. And yet, we all have needs—physical, sexual, emotional, spiritual. So the basic message is, “Your natural way of being is not okay; to be acceptable you must be different from the way you are.”

...

Meditation practices are a form of spiritual reparenting. We are transforming these deeply rooted patterns of inner relating by learning to bring mindfulness and compassion to our life. An open and accepting attention is radical because it flies in the face of our conditioning to assess what is happening as wrong. We are deconditioning the habit of turning against ourselves, discovering that in this moment’s experience nothing is missing or wrong.

Oriah Mountain Dreamer also invites us to embrace acceptance in her poem (and its elaboration in a book of the same name), The Invitation:

I want to know
if you can sit with pain
mine or your own
without moving to hide it
or fade it
or fix it.

And this theme of acceptance is further elaborated in her Prelude to her second book, The Dance:

What if there is no need to change, no need to try and transform yourself into someone who is more compassionate, more present, more loving or wise?

How would this effect all the places in your life where you are endlessly trying to be better?

What if the task is simply to unfold, to become who you already are in your essential nature- gentle, compassionate and capable of living fully and passionately present?

How would this effect how you feel when you wake up in the morning?

What if who you essentially are right now is all that you are ever going to be?

How would this effect how you feel about your future?

What if the essence of who you are and always have been is enough?

There are many other inspiring invitations to practice greater acceptance, but I'll allow myself just one more: Brene Brown's TEDxHouston talk on Wholeheartedness, which evoked a similar response as the Invisibilia episode (I watched the video three times in the span of a few days, each time revealing another layer of deeper emotional resonance and meaning).

The only people who don't experience shame have no capacity for human empathy or connection. No one wants to talk about it, and the less you talk about it, the more you have it. What underpinned this shame, this "I'm not good enough," -- which, we all know that feeling: "I'm not blank enough. I'm not thin enough, rich enough, beautiful enough, smart enough, promoted enough." The thing that underpinned this was excruciating vulnerability. This idea of, in order for connection to happen, we have to allow ourselves to be seen, really seen.

...

when we work from a place, I believe, that says, "I'm enough" ... then we stop screaming and start listening, we're kinder and gentler to the people around us, and we're kinder and gentler to ourselves.

And so, at least in this moment, the kindness and gentleness in me sees the kindness and gentleness in you. Namaste.

Doing Data Science, Helping People Get Jobs @Indeed

Having just marked my 1-year anniversary at Indeed, it occurs to me that I have not yet blogged about my not-so-new job as a data scientist helping people get jobs. In addition to ending a long (7+ month) drought in my blogging practice, I'm also hoping that in sharing a bit about my work at Indeed, I might help more people learn about and get jobs at Indeed's growing Seattle office.

When I tell people I work for Indeed, I get 2 basic types of responses:

• "Cool - I love Indeed!"
• "What's Indeed?"

I don't think I can add much to the first response, except to say that I have found several of my own jobs on Indeed (including my current Data Scientist job at Indeed and my former Principal Scientist job at Nokia Research Center Palo Alto) and helped my daughter find two of her jobs on Indeed, so I love Indeed, too.

To address the second response: Indeed.com is the world's top job site, offering a number of tools to both help people find jobs and help jobs find people (or, more precisely, help employers find employees) - an ideal combination for someone like me with a long-standing passion for making connections ... and helping others make valuable connections.

Indeed helps people get jobs by providing tools to both job seekers and employers. Job seekers can upload or create a resume, search for jobs, receive email alerts when new jobs that match their search criteria are posted, directly apply for some jobs, and track the status of jobs for which they have applied, interviewed or been made offers, all at no cost to job seekers. The company also provides parallel tools for employers to upload jobs, search for resumes, receive email alerts when new resumes that match their search criteria are uploaded.

Employers can enjoy some Indeed services for free. Indeed automatically aggregates millions of job postings from thousands of web sites every day. The site also provides an interface for employers to post jobs directly through Indeed. Paid services include sponsoring jobs to appear in job search results (using a pay-per-click revenue model), contacting job seekers who have uploaded resumes to the site, enabling job seekers to directly apply for jobs through Indeed, and using Indeed's applicant tracking system.

Many things have impressed me during my first year at Indeed, but I'll focus on just a few: mission, measurement, transparency and non-attachment.

The most impressive aspect of Indeed - which was apparent even during the interview process - is the pervasive and relentless focus on the mission of the company: helping people get jobs. I've long been fascinated by the world(s) of work, and it is inspiring to work alongside others who are similarly inspired to help people address a fundamental human need. Just about every internal discussion of a new product or feature at Indeed eventually boils down to the question of "Will this help more people get jobs?"

And the next question is usually "How can we measure the impact?" While a steadily increasing number of users are sharing their stories about finding jobs on Indeed, we don't always know when our users get jobs, so measuring impact often involves various proxies for job-seeking success, but such approximation is a fact of life in most data-driven companies.

Indeed makes extensive use of the Atlassian JIRA software project tracking system for new features, bugs and other issues that arise in the course of software development. Some of the other organizations in which I've worked had cultures of parochialism, secrecy and defensiveness, where critiques were best kept to oneself, or communicated privately. Early on at Indeed, I would often report bugs or make suggestions for improvements via email. After gentle and persistent encouragement, I now report them via JIRA, which - being publicly searchable (within the firm) - increases the possibility for sharing lessons learned. I have yet to encounter an Indeedian who has taken any such feedback personally, or felt so attached to a product feature or segment of code that they weren't willing to consider reviewing and revising it (or allowing someone else to do so) ... and unlike reports I've read about the culture at some other tech companies, I have yet to encounter an asshole at Indeed.

My own work at Indeed currently centers on helping people get jobs by taking greater advantage of the data in the millions of resumes that job seekers have created or uploaded at Indeed. This involves a mix of analyzing, cleaning and provisioning resume data to enhance existing products and inform new products designed to improve search and recommendations for both job seekers and employers.

Over the past several months, I've acted as the chief question answerer in an internal "Resume Q&A" forum we've created to help product managers, data scientists and software engineers better understand and leverage our resume data. Answering these questions has enabled me to practice thoroughly conscious ignorance, offering me numerous opportunities to ask questions of my own, and thereby learn about a broad range of products and processes, as well as various data and code repositories ... and helping forge new connections across them. The work offers me a nice blend of analysis, communication, coding (in Python and Java) and education, a few of my favorite things.

One of the advantages arising from my spiral career path is a user-centered focus I adopted during my years doing user experience research and design. As a practicing data scientist, my UX orientation occasionally helps me trace anomalies in the data back to shortcomings in one or more of the user interfaces or the flow of the user experience across Indeed web services. This UX-oriented data analysis has resulted in at least 2 small, but substantive, changes in the user interface, which I hope has helped more people get jobs.

In addition to regular opportunities to practice my natural inclinations toward instigating and connecting, I've recently started exercising my evangelizing inclinations. I gave a demonstration / presentation on how Indeed can help job seekers to a local job search support group in Bellevue last week, and am hoping to do more evangelizing to job seekers in the future. I am also hoping to start giving more technical presentations on some of the cool things we are doing at Indeed, evangelizing to different audiences, in part, to help us help more data scientists, software engineers, UX designers and researchers, product managers and quality analysts get jobs ... at Indeed, in Seattle and elsewhere.

WTF Economy: Augmentation, Disintermediation and Small Acts of Production

Tim O'Reilly (O'Reilly Media) opened last week's conference on the Next:Economy, aka the WTF economy, noting that "WTF" can signal wonder, dismay or disgust. I experienced all three reactions at different times during the ensuing two-day "investigation into the potential of emerging technologies to remake our world for the better". I attended the conference because I have long been interested in the nature and meaning of work, and now that I work for Indeed, I am particularly interested in how we can remake our world by reimagining how the world works for the better.

The conference presentations and discussions, curated by Tim and his co-organizers Steven Levy (Backchannel) and Lauren Smiley (Medium), provided many interesting insights, experiences and provocations. If I had to choose three top themes that emerged for me, they would be

• Augmentation vs. automation: technology designed to assist human workers by taking over some of their tasks vs. technology designed replacing human workers by taking over all of their tasks
• The disintermediation of creative work: the vast array of tools and resources available for creative people to pursue their (our?) passions while earning a sustainable income outside of the constraints of a traditional job or company
• Small is beautiful: the growing number of platforms for designing and creating products and/or offering services at small scales that can enrich the lives of the producers, consumers and society at large

I'm sharing - and shortening - my conference notes here, as I tend to understand and remember what I learn better when I re-process them for potential public consumption. I've also compiled a Twitter list including all the Next:Economy speakers I could find. Kevin Marks compiled and shared a much more extensive collection of notes from day 1 and day 2 of the conference, and a Pinboard storify offers an alternative perspective.

No ordinary disruption

James Manyika (McKinsey Global Institute) shared so many interesting facts, figures and forecasts that I had a hard time keeping up. The following are among the nuggets I was able to capture:

• The consuming class (those who live above the subsistence level) will rise from 23% of the population to 50% in 2025
• The last 50 years of year-over-year 3.5% average global GDP growth were fueled by the combined growth of labor (1.7%) and productivity (1.8%); with labor supply expected to peak in 2050, productivity growth will have to increase 80% to maintain overall growth rates
• Labor markets don't work very well, producing massive shortages of workers with the right skills in the right places; digital platforms for helping increasing the quality of matches and decreasing job search times may help improve labor markets as labor supply declines
• If all countries were to improve gender parity to level achieved by their "best in region" neighbors, we could add as much as $12 trillion (11%) to annual 2025 GDP
• 45% of tasks could be automated, but only 5% of jobs can be completely automated; up to 30% of tasks in 60% of jobs could be automated, redefining occupations and skills needed
• We need to change our mindset from jobs to work, and from wages to income

From self-driving cars to retraining humans

Sebastian Thrun (Co-Founder and CEO, Udacity) offered an interesting perspective on human learning vs. machine learning: when a person makes a mistake (e.g., causes an automobile accident), that person learns, but no one else learns; when a robot (e.g., a self-driving car) makes a mistake, all other robots can learn. He said Udacity's strategy is to develop nano-courses and nano-degrees in response to industry demand for specific skills, to help more people get unstuck more quickly. He also recommends that everyone become an Uber driver for a day.

The future of personal assistants

Alexandre Lebrun (Head, Wit.ai, Facebook) said that Facebook's M personal assistant is designed to interact with customers as far as it can go, and then observe human customer service representatives ("trainers") when they intervene so it can learn how to handle new situations. Adam Cheyer (Co-founder and VP of Engineering, Viv Labs) talked about exponential programming in which an application can write code to extend itself; in the case of the Viv personal assistant, when you make a request, Viv writes a custom program at that moment to respond to that request.

Will robots augment us or rule us?

John Markoff (Journalist, New York Times) identified two divergent trajectories that emerged at Stanford around 1962: John McCarthy's lab focused on autonomous systems (Artificial Intelligence or AI) while Doug Englebart's lab focused on augmenting human intelligence (Human Computer Interaction or HCI), and noted that system designers have a choice: design people into the system or design people out. Jerry Kaplan (Visiting Lecturer, and Fellow at The Stanford Center for Legal Informatics, Stanford University) said the oft-cited [especially at this conference] 2013 Oxford study on the future of employment, which warned that 47% of total US employment is at risk of being automated was intellectually interesting but critically flawed; his view is that tasks will continue to be automated - as they have for 200 years - but not jobs, except for jobs whose tasks can all be automated (consistent with James Manyika's earlier forecast).

"Knowledge Work": No longer safe from automation

Kristian Hammond (Chief Scientist, Narrative Science) began with a bold claim about Quill, his company's software that uses templates to automatically construct narratives (text) to explain structured data: "If your data and their analysis have meaning, Quill can transform that into language". Since one might characterize everything in the world that humans attend to as some mix of data and analysis, I'm not sure exactly what that claims means, but I do understand - and like - his more modest claim that Quill specializes in small audiences ("writing for one"). I was also impressed with some of the examples of pilot projects he gave, including narrativizations of MasterCard's narrativization of targeted recommendations for small business owners, a portfolio fund manager's quarterly report (which typically takes a month to produce) and web site performance reporting. Echoing the theme of augmenting human intelligence that pervaded many presentations, he proposed that "If anyone has the word 'analyst' in their job title, something like Quill is going to be working with them at some point". And ending on yet another provocative note, he observed that "we need data scientists everywhere, but they don't scale" and suggested "'data scientist' is the sexiest job of the 21st century; it's the next job we're going to automate".

The Kickstarter economy

Yancey Strickler (Co-Founder & CEO, Kickstarter) presented some examples of how his crowd funding site has redefined hardware design as an artful medium, facilitating small-scale manufacturing of product lines with 150-5000 units and enabling creative people to be more independent and move easily from product to product. He also articulated three guiding principles for the WTF economy:

1. DON"T SELL OUT (but still survive); creative people often feel guilty charging money for work they do, but they need to earn enough to support themselves and their families
2. BE IDEALISTIC: you don't have to buy into the money monoculture and its new rules for competition based on anxiety, paranoia, disruption and war (Kickstarter is now a Public Benefit Corporation)
3. IT'S HARDER, BUT IT'S EASIER: it's harder to measure success without an exclusive focus on financial profits, but it's easier to make decisions based on more human-centered principles

The small scale factory of the future

Limor Fried (hacker, slacker, code cracker, Adafruit Industries) gave the most engaging remote presentation I've ever witnessed from her 80-person, $40M open-source hardware company's factory in New York. She articulated a particularly pithy description of Adafruit, which has produced nearly 900 tutorials -"we're an education company, with a gift shop at the end" - and offered an endearing story of a 6-year-old girl who regularly watches the Adafruit "Ask an Engineer" show, which features so many women engineers that the girl asked "Daddy, are there any men engineers?"

One, two, three, boom!

Mark Hatch (CEO, TechShop) launched into his presentation by informing us "I love revolutions" such as the one that was ignited when "the middle class got access to the tools of the industrial revolution", and "I'm a former Green Beret, and I love blowing things up". He proceeded to give us a whirlwind tour of his multifaceted response to the question "Has anything serious come out of the maker movement?", inviting the audience to share his enthusiasm for each example of a product - and lifestyle business - created bymembers of his TechShop hackerspaces by shouting "Boom!". The examples included an electric motorcycle, a jet pack, a desktop diamond manufacturing device, a laser-cutting cupcake topper, an underwater robot, as well as more well-known companies that started as a TechShop project such as Square, Solum and DripTech (the latter 2 being among the 5 top agricultural startups of 2014).

Real R&D is hard

Saul Griffith (CEO, Otherlab) shared highlights from two seminal 1945 essays by Vannevar Bush: As We May Think, which presaged the Internet, speech recognition and online knowledge sources (among other things) and the lesser-known report, Science, the Endless Frontier, which effectively transformed and transitioned research from small labs to large universities. With Google''s R&D budget of $9B now rivaling the combined funding of the National Science Foundation ($6B) and the Defense Advanced Research Project Agency (DARPA, $3B), we are in the midst of a transition of research from universities toward industry. Smaller scale industry research labs, such as Otherlab) have produced innovations that include drones used for wind power generation, inexpensive and durable trackers for solar panels, cheap actuators that can be used on a [slow] walking inflatable elephant robot, pneubotic (air-filled / air-powered) robots that can lift their own weight and minimize risks that their heavier counterparts pose to human collaborators, and soft exoskeletons that can enable anyone to run faster with less energy.

Why services aren't enough

Jeff Immelt (Chairman and CEO, General Electric) noted that industrial productivity growth has been declining from 4% (2006-2011) to 1% (2011-2015), but that reframing industrial products as data platforms (e.g., a locomotive today is a rolling data center with 600+ sensors) may open up opportunities for new products and productivity gains.

Workplace monitoring, algorithmic scheduling, and the quest for a fair workweek

Esther Kaplan (Editor, The Investigative Fund) chaired a panel discussion about what it is like to live (and work) in the reality of "the hyper-lean, electronically scheduled labor force", focusing primarily on retail jobs, which make up 10% of all US jobs.

Darrion Sjoquist (Starbucks barista, Working Washington) shared stories of growing up with a mother who worked for Starbucks, in which the entire family and was adversely affected by the unpredictability arising from 5-day advance schedule notices and clopenings (a closing shift followed by an opening shift the next day), making it difficult to commit to clubs or other extra-curricular school activities, as well as stories from his own experience as a Starbucks barista, in which co-workers and customers are at risk from sick employees doing their best to operate in a work regimen that is so lean that they cannot afford to take sick days.

Carrie Gleason (Director, Fair Workweek Initiative Center for Popular Democracy) noted that involuntary part-time employment more than doubled between 2008-2010, cited research on the instability created by schedule unpredictability among early career workers in the US labor market and said that workers are trying to minimize the instability by setting up shift-swapping groups on Facebook. She also announced a new Fair Workweek Initiative designed to "provide working families with stable employment, a livable income, and family-sustaining scheduling".

Charles DeWitt (Vice President, Business Development, Kronos) said that Walmart ushered in an era in which labor became "a big bucket of cost and compliance issues", and people became an asset to be optimized. Schedule stability was not factored into workforce management software, but it could be. Citing the annual Gallup Employment Engagement Survey, and Gallup's estimate that disengaged employees cost the US $450-550B in lost productivity each year, he proposed that linking scheduling practices to employee engagement metrics may be a good way to promote greater stability.

Does on demand require independent contractors?

Leah Busque (Founder, TaskRabbit) framed TaskRabbit as service networking vs. social networking, which now includes 30K taskers (people who perform tasks for pay) in 21 cities, who make an average of $35/hour and $900/month across all locations, with 10% of taskers working full-time (via TaskRabbit). The service was first launched in Boston in 2008, where the large population of students was expected to provide plenty of candidate taskers; instead, early taskers tended to be stay-at-home moms, retired people and young professionals. Leah noted that "any business with a platform with 2 sides of a market has to make both sides successful" ... however, she did not recommend that everyone sign up to become a tasker for a day.

What's it like to drive for Uber or Lyft?

Eric Barajas (Driver, Uber), Jon Kessler (Driver, Lyft, former cab driver) and Kelly Dessaint (Driver, National Veterans Cab, former Uber and Lyft driver) provided a lively exchange of insights and experiences regarding the costs and benefits for driving for different companies. I don't know whether they would want their income shared publicly (outside the conference), but it appeared that Kelly, the cab driver, is faring better than the others in terms of higher revenue (due, in part, to larger and more frequent tips) and lower expenses (a daily gate fee of $111-121). Uber and Lyft drivers are responsible for gas, vehicle maintenance and insurance (Uber and Lyft only cover insurance only while drivers have paying passengers in the vehicle), and, in some cases, a lease; they face the perpetual prospect of being "deactivated" due to a customer complaint, and so have relatively little autonomy once they pick up a passenger. Kelly noted that taxi drivers are a community - "we look at each other, we nod" - and warned "the worst drivers in SF are tourists" and "most Uber/Lyft drivers are tourists" (because they live outside the city); he offered an interesting counter-perspective to what I've heard from the relatively few cab drivers I've traveled with recently.

The changing nature of work

Esko Kilpi (Managing Director, Esko Kilpi Company) observed that "People are not clever, people have never been clever, and people will never be clever" and so we create and use technology to compensate. He argued that the work systems we have are broken, because they are based on artificial scarcity and "wrong ideas about who human beings are", failing to take into account different situations with different demands. While work has always been about solving problems; it used to be that your boss told you the problem you were supposed to solve, but increasingly, defining the problem is itself part of the work, and so work and learning are inextricably linked. I highly recommend his recent Medium post on The New Kernel of On-Demand Work.

What's the investment opportunity?

Simon Rothman (Partner, Greylock Partners), Gary Swart (Venture Partner, Polaris Partners) and James Cham (Bloomberg Beta [whose homepage is on GitHub (!)]) shared a number of interesting insights; unfortunately, I missed the introductions and so cannot offer precise attribution, but here are some of the highlights shared during this session:

• Just because the customer model is appealing doesn't mean the business model will work
• A lot of platforms that call themselves marketplaces are actually managed services
• Well intermediated marketplaces feel like a service
• Regulation is a historical artifact, attempting to project the past into the present / future
• Silicon Valley is the QA department for the rest of the world

Supporting workers in the on-demand economy

Nick Grossman (General Manager, Union Square Ventures) presented his views on the deconstruction of the firm, or "the jobs of a job": brand, income, customers, taxes and administration, benefits and insurance, facilities and equipment, scheduling, community, training. More and more of these elements can be found outside of traditional companies, enabling Individual workers to become, in effect, networked micro-firms.

Creating better teams

Stewart Butterfield (Co-founder and CEO, Slack) shared his view on the evolution of objects around which computer applications are based: applications, documents and (now) relationships. Slack has 1.7M users, including 1M paid subscribers, and for 3+ years was produced unselfconsciously as a tool to support lateral transparency while the team worked on what they thought would be their primary product (a game), until the tool itself emerged as a useful product in its own right.

Tax and accounting tools for the franchise of one

Brad Smith (President and CEO, Intuit) talked about the difficulties of quantifying "self-employment" because the term can be defined in so many ways. 78% of US Intuit users who are self-employed have 3 or more sources of income. He sees the recent extension of myRA (my Retirement Account) eligibility from small- to medium-sized businesses (SMBs) to self-employed people as a more positive step than most of the other reviews I've read.

Flexibility needed: Not just for on-demand workers

Anne-Marie Slaughter (President and CEO, New America) noted that "having a job that allows you to support your family is an essential ingredient in a personal life narrative" and that supporting your family means more than earning income: workers need the flexibility to care for family members. In discussing generous parental leave policies announced recently by some firms, she argued that you have to change the culture, not just the policy, and so extended parental leave won't matter much unless the men in senior leadership positions take advantage of the new policies.

Conference dinner

There were two separate talks presented by Code for America fellows: one has avoided signing a rental lease or owning a home through serial AirBnB stays in each place he's worked for a number of years now; another went on food stamps in his effort to better understand the challenges faced by food stamp recipients in California. Unfortunately, I did not take notes during either talk, and now cannot even remember the names of the presenters.

Humans need not apply? Not so fast!

Nick Hanauer (Second Avenue Partners) offered a number of observations and insights:

• Some amount of economic inequality is good (healthy incentives), but too much is bad
• We think of prosperity as money, but this is wrong
• What matters is the accumulation of solutions to human problems
• Not GDP, but the rate at which we solve these problems
• "How we improve our lives" is the point of the economy
• Revolutions, like bankruptcies, come gradually, and then suddenly (see also: The Pitchforks are coming .. for us Plutocrats)
• "Trickle-down economics is an intimidation tactic masquerading as an economic theory" [my favorite sound bite from the conference]

Managing talent in the networked age

Reid Hoffman (Co-Founder & Executive Chairman, LinkedIn; Partner, Greylock Partners) and Zoë Baird (CEO and President, Markle Foundation) announced the Markle Rework America initiative: "A 21st Century Digital Labor Market for Middle-Skill Job Seekers", designed to boost the signaling and improve the matching among employers, workers and educators. The initiative partners include LinkedIn, Arizona State university and edX, as well as regional partners in Colorado and Phoenix (the regions where the program is initially being rolled out). Reid wrote an interesting critique of higher education, Disrupting the Diploma, a few years ago, highlighting the need to "make certification faster, cheaper, and more effective". Zoe suggested that we need to create boot camps for other kinds of training to support other parts of the market beyond the technology sector, and explore new methods of credentialing beyond the traditional degrees granted by colleges and universities (which collectively make up the third largest lobbying group in Washington DC).

Exponential teaching

Kimberly Bryant (Founder, Black Girls CODE) championed exponential teaching: "If you teach one girl, she will naturally turn around and teach five, six, or 10 more”. Teaching black girls to code is a promising effort to counteract the effects of 45% of black women aged 25 and over having no high school diploma. The grass-roots organization has created 10 chapters in 4 years, and she invoked the optimism of Grace Lee Boggs with respect to the program's prospects: "I believe we are at the point now, in the US, where a movement is starting to emerge".

Matching workers with opportunities at high velocity

Stephane Kasriel (CEO, Upwork) talked about how UpWork ("Tinder for work") matches freelance knowledge workers (primarily software developers) to remote work opportunities, shortening the typical job search from 3-4 weeks to 1-2 days. With the steadily declining half-life of any skill (especially in the technology sector), Stephane declared "the resume is dead", and "all we want to do is reduce all the time people waste writing long form job descriptions and long form job proposals". To match workers with work, Upwork utilizes three different machine learning models to determine

• Who is qualified?
• Who is interested?
• Who is available?

Upwork has 150K work descriptions and 250K worker profiles, with 5K workers signing up every day. Stephane said only about 2% of new workers get jobs right away, and one of the challenges for Upwork ("the celebrity agent of the freelancers") is to effectively manage talent at different stages of their pipeline (newcomers, rising stars and established workers). Stephane suggested that industry is not doing a good job of communicating data about needed skills back to academia. Having looped in and out of academia a few times myself, I would suggest that academia is not configured in a way that facilitates rapid response to changing educational needs, and so other learning channels will likely be needed to support the continuous cycle of nano-jobs, nano-skills and nano-degrees in the future.

Work rules: Lessons from Google's success

Laszlo Bock (Senior Vice President of People Operations, Google) said an internal study revealed there is little predictive value in determining the probability of success based on which school a candidate attended; the most predictive value was found in a sample work test, in which a candidate performs the type of work associated with the job function, and the second most predictive value was in cognitive ability, which can be assessed via structured interview questions (give me an example of a time when you did X). Google does not look only for superstars, since many superstars don't perform as well when they switch companies, but also looks for team players who improve the performance of everyone around them (like basketball player Shane Battier). Among their efforts to promote diversity, they have implemented an unconscious bias training program, and a Google in Residence program in which Google engineers are embedded in a handful of historically black colleges and universities where they advise on curriculum matters and mentor students. Google also conducts an anonymous survey of 8 management attributes to provide specific feedback (with no penalties) to managers; they have found that average favorability ratings increase every cycle, and that a 2-hour management program can help managers improve by 6-7 points.

Intelligent agents, augmented reality, and the future of productivity

Satya Nadella (CEO, Microsoft) posited the agent as the third runtime model (after the PC operating system and the browser), and talked about some of Microsoft's work in speech recognition, augmented reality and machine learning. He described augmentation as a "race with the machine", rather than a race against the machine (which may be a more apt description of automation).

Augmented reality in the factory

Brian Mullins (Founder and CEO, DAQRI) asked "What if you could put on a helmet and do any job?" and proceeded to present a number of applications of their augmented reality Smart Helmet: digitizing analog devices (e.g., gauges), providing thermal vision and improving "cognitive literacy". He went into some detail - including a video - on one deployment in partnership with Hyperloop using the helmet in a steel mill. I can't find that video, but found another video from EPRI that provides a better sense of how the Smart Helmet works.

How augmented workers grow the market

David Plouffe (Chief Advisor and Member of the Board, Uber) posed what he calls the central government question: How do we get more income to more people? He said asking how Uber is affecting the taxi market is the wrong question; instead, we should be looking at how Uber is affecting a multi-modal ecosystem, noting that fewer millennials are choosing to own cars and expanding transportation options can be crucial in helping people escape poverty. Uber has 400K drivers, but 100K drive only a few times a month and 50% of drivers drive < 10 hours / week. During the Q&A session, Eric Barajas, the Uber driver who had appeared on the previous day's panel, asked about what Uber can do to better support drivers who are driving full-time, and was invited to come talk to the Uber SF office about it (I hope he doesn't suffer any adverse consequences from speaking up).

Reinventing healthcare

Lynda Chin (Director, Institute for Health Transformation) talked about the Oncology Expert Adviser, a healthcare application based on IBM Watson. As is so often the case when I read about Watson in healthcare, I love the idea of a system that can take in all kinds of input - text analysis of articles from the medical literature as well as individual patient charts, diagnoses of past cases by human experts - and assist doctors in diagnosis and treatment. Unfortunately, as is also so often the case when I hear or read about Watson in healthcare, I don't get a clear idea of what it can actually do yet.

Policy action recommendations for the 21st century economy

In this panel discussion, Felicia Wong (President and CEO, Roosevelt Institute) championed campaign finance reform, reversing the financialization of the economy and making massive investment in human capital; Neera Tanden (President, Center for American Progress) noted that the US is relatively unique among leading industrial countries in the downward pressure on wages and pointed toward Australia and Canada as better models for achieving median income growth through higher rates of unionization (30% vs. 6%), more equitable education (higher education is [nearly] free) and relatively smaller and more regulated financial sectors; Zoë Baird (CEO and President, Markle Foundation) recommended the adoption of policies that will help SMBs better reach global markets.

Rewiring the US labor market

Byron Auguste (Managing Director, Opportunity@Work) claims that "talent is far more evenly distributed than opportunity or money" and that the labor market is "a wildly inefficient 'efficiency' market", pointing to credential creep (e.g., only 20% of current administrative assistants have bachelor degrees and yet 65% of administrative assistant job descriptions require them), Okun's Law (the correlation between increased unemployment and reduced GDP), a quit rate that is down 28% since 2010 and a record number (5.8M) of unfilled jobs. The White House TechHire initiative includes a national employer network and an open platform to provided better training for and access to information technology jobs in 31 cities for some of the segments of the US population that are not being well served by the current labor market:

• 30-40M college goers who did not graduate
• 15-20M caregivers limited in ability to work for pay
• 10-15M experienced skilled older workers needing to re-tool
• 1.5M veterans who are unemployed or entering the workforce soon
• 6M disconnected youth

Worker voice in the 21st century

Jess Kutch (Digital Strategist and Co-founder, Coworker.org) and Michelle Miller (Co-founder, Coworker.org) addressed the conference via video, providing examples of workers speaking up and instigating changes in companies, using their coworker.org petition campaign platform. One notable example is the successful effort by Starbucks baristas to change the company policy barring visible tattoos. Perhaps more relevant, given other sessions at the conference, are petition campaigns to urge Starbucks to give employees a fair workweek and to urge Uber to give consumers the option of adding a tip to all Uber fares.

Reinventing the labor union

Liz Shuler (Secretary-Treasurer / CFO, AFL-CIO) talked about some of the challenges facing unions ("the original disrupters" of the workplace) and unionization efforts. Andy Stern (Senior Fellow, Columbia University; former President, SEIU) talked about the strategic inflection point we are approaching with respect to automation's potential impact on work and the long-standing connection between work and income. He suggested that now is an ideal time to consider universal basic income, which guarantees everyone a basic level of income by having government give money directly to those in poverty, rather than via special programs such as food stamps and earned income tax credits that are burdensome for everyone involved. "If you want to end poverty give people money".

Portable benefits and the "shared security account"

Laura Tyson (Professor & Director of Institute for Business and Social Impact, Haas School of Business at UC Berkeley) spoke with Nick Hanauer (Second Avenue Partners) and David Rolf (President, SEIU 775) about some of the ideas they raised in an inspiring article on Shared Security, Shared Growth, in which they argue for decoupling benefits from specific jobs and attaching them to the workers, a separation which will become increasingly important as workers increasingly work multiple jobs (or tasks). Among the many interesting topics discussed was the incentives for CEOs to buy back stock (vs. incentives for programs that might benefit workers), which led me to an HBR article on profits without prosperity.

Reinventing public transportation

Logan Green (CEO, Lyft) talked about some of the ways that Lyft is trying to support its drivers. Although 78% of Lyft drivers drive < 15 hours / week, they offer a power driver bonus, giving increasing amounts of commission back as drivers drive more. The Lyft app offers passengers the option to include a tip, though Tim O'Reilly (who was interviewing Logan) said that a Lyft driver told him that only 20% of passengers leave a tip. Toward the end of the conversation, Logan noted that the least profitable runs in public transportation are the most expensive, and that some kind of public/private partnership might enable Lyft to complement public transportation.

A people-centered economy

Chad Dickerson (CEO, Etsy) talked about how Etsy embraces the idea that small is beautiful. In addition to enabling individual artisans to sell their creations, Etsy is reimagining manufacturing, allowing Etsy users to register as manufacturers if they want to work with other Etsy users to help create their products. There are also self-organized teams of Etsy sellers around the world, and he gave an example where Italian sellers were encouraging prospective buyers to buy things produced from Greek sellers during the Greece financial crisis. He also said that Etsy has become a Public Benefits Corporation.

The good jobs strategy

Zeynep Ton (Associate Professor, MIT Sloan School of Management) presented her research into the financial success enjoyed by companies that embrace human-centered systems and provide jobs with meaning and dignity, offering the following principles: operate with some slack (not too lean), offer less (fewer products), cross train, standardize and empower. Dan Teran (co-founder, Managed by Q) said he recognized early on that "if we wanted to have the best employees, we had to be the best employer"; engineers at the company go out and clean an office the first week, so they can better understand the tasks and environment in which the cleaners who work for the company operate. Palak Shah (Social Innovations Director, National Domestic Workers Alliance) noted that "in a way you can consider domestic workers [nannies, caregivers, cleaners] as the original gig workers" and presented the Good Work Code, "an overarching framework of 8 simple values that are the foundation of good work".

Enable people, and they will amaze you

Evan Williams (CEO, The Obvious Corporation, and founder of Medium and Blogger), whose first tech job was with O'Reilly Media, said the thing that got him excited about the Internet was the idea of knowledge exchange. One of the motivations behind Medium was that comments on blogs are not on the same level as the posts; on Medium, replies are posts, and so all commentary is on the same level. Noting that "what you measure gets rewarded", Medium measures time on a page, not [just] page views or unique visitors. In the Q&A session, someone asked "how do you think about the future". Ev replied "I just listen to Tim" ... and then Tim replied "I look for people who are passionate about what they do".

The conference sessions offered a compelling collection of people who are passionate about what they do, and much of what many of the speakers do is helping other people find ways to exercise their passions in what they do ... and a recursive promulgation of passion seems to be as good a note as any on which to end this post.

Notes from #PyData Seattle 2015

I was among 900 attendees at the recent PyData Seattle 2015 conference, an event focused on the use of Python in data management, analysis and machine learning. Nearly all of the tutorials & talks I attended last weekend were very interesting and informative, and several were positively inspiring. I haven't been as excited to experiment with new tools since I discovered IPython Notebooks at Strata 2014.

I often find it helpful to organize my notes after attending a conference or workshop, and am sharing these notes publicly in case they are helpful to others. The following is a chronological listing of some of the highlights of my experience at the conference. My notes from some sessions are rather sparse, so it is a less comprehensive compilation than I would have liked to assemble. I'll also include some links to some sessions I did not attend at the end.

Python for Data Science

Joe McCarthy, Indeed, @gumption

This was my first time at a PyData conference, and I spoke with several others who were attending their first PyData. Apparently, this was the largest turnout for a PyData conference yet. I gave a 2-hour tutorial on Python for Data Science, designed as a rapid on-ramp primer for programmers new to Python or Data Science. Responses to a post-tutorial survey confirm my impression that I was unrealistically optimistic about being able to fit so much material into a 2-hour time slot, but I hope the tutorial still helped participants get more out of other talks & tutorials throughout the rest of the conference, many of which presumed at least an intermediate level of experience with Python and/or Data Science. As is often the case, I missed the session prior to the one in which I was speaking - the opening keynote - as I scrambled with last-minute preparations (ably aided by friends, former colleagues & assistant tutors Alex Thomas and Bryan Tinsley).

Scalable Pipelines w/ Luigi or: I’ll have the Data Engineering, hold the Java!

Jonathan Dinu, Galvanize, @clearspandex

Running and re-running data science experiments in which many steps are repeated, some of which are varied (e.g., with different parameter settings), and several take a long time are all part of a typical data science workflow. Every company in which I've worked as a data scientist has rolled their own workflow pipeline framework to support this process, and each homegrown solution has offered some benefits while suffering from some shortcomings. Jonathan Dinu demonstrated Luigi, an open source library initially created by Spotify for managing batch pipelines that might encompass a large number of local and/or distributed computing cluster processing steps. Luigi offers a framework in which each stage of the pipeline has input, processing and output specifications; the stages can be linked together in a dependency graph which can be used to visualize progress. He illustrated how Luigi could be used for a sample machine learning pipeline (Data Engineering 101), in which a corpus of text documents is converted to TF-IDF vectors, and then models are trained and evaluated with different hyperparameters, and then deployed.

Keynote: Clouded Intelligence

Joseph Sirosh, Microsoft, @josephsirosh

Joseph Sirosh sprinkled several pithy quotes throughout his presentation, starting off with a prediction that while software is eating the world, the cloud is eating software (& data). He also introduced what may have been the most viral meme at the conference - the connected cow - as a way of emphasizing that every company is a data company ... even a dairy farm. In an illustration of where AI [artificial intelligence] meets AI [artificial insemination], he described a project in which data from pedometers worn by cows boosted estrus detection accuracy from 55% to 95%, which in turn led to more successful artificial insemination and increased pregnancy rates from 40% to 67%. Turning his attention from cattle to medicine, he observed that every hospital is a data company, and suggested that Florence Nightingale's statistical evaluation of the relationship between sanitation and infection made her the world's first woman data scientist. Sirosh noted that while data scientists often complain that data wrangling is the most time-consuming and challenging part of the data modeling process, that is because deploying and monitoring models in production environments - which he argued is even more time-consuming and challenging - is typically handed off to other groups. And, as might be expected, he demonstrated how some of these challenging problems can be addressed by Azure ML, Microsoft's cloud-based predictive analytics system.

The past, present, and future of Jupyter and IPython

Jonathan Frederic, Project Jupyter, @GooseJon

IPython Jupyter Notebooks are one of my primary data science tools. The ability to interleave code, data, text and a variety of [other] media make the notebooks a great way to both conduct and describe experiments. Jonathan described the upcoming Big Split(tm), in which IPython will be separated from Notebooks, to better emphasize the increasingly language-agnostic capabilities of the notebooks, which [will soon] have support for 48 language kernels, including Julia, R, Haskell, Ruby, Spark and C++. Version 4.0 will offer capabilities to

• ease the transition from large notebook to small notebooks
• import notebooks as packages
• test notebooks
• verify that a notebook is reproducible

As a former educator, a new capability I find particularly exciting is nbgrader, which uses the JupyterHub collaborative platform, providing support for releasing, fetching, submitting and collecting assignments. Among the personally most interesting tidbits I learned during this session was that IPython started out as Fernando Perez' "procrastination project" while he was in PhD thesis avoidance mode in 2001 ... an outstanding illustration of the benefits of structured procrastination.

Deep Learning with Python: getting started and getting from ideas to insights in minutes

Alex Korbonits, Nuiku, @korbonits

Deep Learning seems to be well on its way toward the peak of inflated expectations lately (e.g., Deep Learning System Tops Humans in IQ Tests), Alex Korbonits presented a number of tools for and examples of Deep Learning, the most impressive of which was AlexNet, a deep convolutional neural network developed by another Alex (Alex Krizhevsky, et al) that outperformed all of its competitors in the LSVRC 2010 ImageNet competition (1.3M high-res images across 1000 classes) by such a substantial margin that it changed the course of research in computer vision, a field that had hitherto been dominated by hand-crafted features refined over a long period of time. Alex Korbonits went on to demonstrate a number of Deep Learning tools & packages, e.g., Caffe and word2vec, and applications involving scene parsing and unsupervised learning of high-level features. It should be noted that others have taken a more skeptical view of Deep Learning, and illustrated some areas in which there's still a lot of work to be done.

Deep Learning with Python (PyData Seattle 2015) from Alexander Korbonits

Jupyter for Education: Beyond Gutenberg and Erasmus

Paco Nathan, O’Reilly Media, @pacoid

One of the most challenging aspects of attending a talk by Paco Nathan is figuring out how to bide my time between listening, watching, searching for or typing in referenced links ... and taking notes. He is a master speaker, with compelling visual augmentations and links to all kinds of interesting related material. Unfortunately, while my browser fills up with tabs during his talks, my notes typically end up rather sparse. In this talk, Paco talked about the ways that O'Reilly Media is embracing Jupyter Notebooks as a primary interface for authors using their multi-channel publishing platform. An impressive collection of these notebooks can be viewed on the O'Reilly Learning page. Paco observed that the human learning curve is often the most challenging aspect to leading data science teams, as data, problems and techniques change over time. The evolution of user expertise, e.g., among connoisseurs of beer, is another interesting area involving human learning curves that was referenced during this session.

Counterfactual evaluation of machine learning models

Michael Manapat, Stripe, @mlmanapat

Fraud detection presents some special challenges in evaluating the performance of machine learning models. If a model is trained on past transactions that are labeled based on whether or not they turned out to be fraudulent, once the model is deployed, the new transactions classified by the model as fraud are blocked. Thus, the transactions that are allowed to go through after the model is deployed may be substantially different - especially with respect to the proportion of fraudulent transactions - than those that were allowed before the model was deployed. This makes evaluation of the model performance difficult, since the training data may be very different from the data used to evaluate the model. It also complicates the training of new models, since the new training data (post model deployment) will be biased. Michael Manapat presented some techniques to address these challenges, involving allowing a small proportion of potentially fraudulent transactions through and using a propensity function to control the "exploration/exploitation tradeoff".

Counterfactual evaluation of machine learning models from Michael Manapat

Keynote: A Systems View of Machine Learning

Josh Bloom, UC Berkeley & wise.io, @profjsb

In the last keynote of the conference, Josh Bloom shared a number of insights about considerations often overlooked by data scientists regarding how data models fit into the systems into which they are deployed. For example, while data scientists are often concerned with optimizing a variety parameters in building a model, other important areas for optimization are overlooked, e.g., hardware and software demands of a deployed model (e.g., the decision by Netflix not to deploy the model with the highest score in the Netflix Prize), the human resources required to implement and maintain the model, the ways that consumers will [try to] interpret or use the model, and the direct and indirect impacts of the model on society. Noteworthy references include a paper by Sculley, et al, on Machine Learning: The High Interest Credit Card of Technical Debt and Leon Bottou's ICML 2015 keynote on Two Big Challenges of Machine Learning.

PyData 2015 Keynote: "A Systems View of Machine Learning" from Joshua Bloom

NLP and text analytics at scale with PySpark and notebooks

Paco Nathan, O'Reilly Media, @pacoid

Once again, I had a hard time keeping up with the multi-sensory inputs during a talk by Paco Nathan. Although I can't find his slides from PyData, I was able to find a closely related slide deck (embedded below). The gist of the talk is that many real-world problems can often be represented as graphs, and that there are a number of tools - including Spark and GraphLab - that can be utilized for efficient processing of large graphs. One example of a problem amenable to graph processing is the analysis of a social network, e.g., contributors to open source forums, which reminded me of some earlier work by Weiser, et al (2007), on Visualizing the signatures of social roles in online discussion groups. The session included a number of interesting code examples, some of which I expect are located in Paco's spark-exercises GitHub repository. Other interesting references included TextBlob, a Python library for text processing, and TextRank, a graph-based ranking model for text processing, a paper by Mihalcea & Tarau from EMNLP 2004.

Graph Analytics in Spark from Paco Nathan

Pandas Under The Hood: Peeking behind the scenes of a high performance data analysis library

Jeffrey Tratner, Counsyl, @jtratner

Pandas - the Python open source data analysis library - may take 10 minutes to learn, but I have found that it takes a long time to master. Jeff Tratner a key contributor to Pandas - an open source community he described as "really open to small contributors" - shared a number of insights into how Pandas works, how it addresses some of the problems that make python slow, and how the use of certain features can lead to improved performance. For example, specifying the data type of columns in a CSV file via the dtype parameter in read_csv can help pandas save space and time while loading the data from the file. Also, the Dataframe.append operation is very expensive, and should be avoided wherever possible (e.g., by using merge, join or concat). One of my favorite lines: "The key to doing many small operations in Python: don’t do them in Python"

Mistakes I've Made

Cameron Davidson-Pilon, Shopify, @cmrn_dp

While I believe that there are no mistakes, only lessons, I do value the relatively rare opportunities to learn from others' lessons, and Cameron Davidson-Pilon (author of Probabilistic Programming & Bayesian Methods for Hackers) shared some valuable lessons he has learned in his data science work over the years. Among the lessons he shared:

• Sample sizes are important
• It is usually prudent to underestimate predictions of performance of deployed models
• Computing statistics on top of statistics compounds uncertainty
• Visualizing uncertainty is a the role of a statistician
• Don't [naively] use PCA [before regression]

Among the interesting, and rather cautionary, references:

• The Most Dangerous Equation, Howard Wainer, American Scientist (2007)
• How A Single A/B Test Increased Conversions by 336%, Dustin Sparks

There were a few sessions about which I read or heard great things, but which I did not attend. I'll include information I could find about them, in the chronological order in which they were listed in the schedule, to wrap things up.

Testing for Data Scientists

Trey Causey, Dato, @treycausey

Learning Data Science Using Functional Python

Joel Grus, Google, @joelgrus

Code + Google docs presentation (can't figure how to embed)

Big Data Analytics - The Best of the Worst : AntiPatterns & Antidotes

Krishna Sankar, blackarrow.tv, @ksankar

Big Data Analytics - Best of the Worst : Anti-patterns & Antidotes from Krishna Sankar

Python Data Bikeshed

Rob Story, Simple, @oceankidbilly

GitHub repo

Low Friction NLP with Gensim

Trent Hauck, @trent_hauck

Slides [PDF]

[Update, 2015-08-05: the PyDataTV YouTube channel now has videos from the conference]