|
|
By Designdialogues, on August 24th, 2010
I wonder why scientists, who require significant levels of validation in work in their own disciplines, make rather un-scientific analyses about scientific practices. In this case, paper publishing and peer review.
Peer review, the blind circulation of research manuscripts among a community of reviewers for assessing editorial and content fit to a journal topic, has been breaking down for some time. There are many reasons why this is the case, including work intensification, increasing scientific specialization, and increasing number of journals. Yes these root cause factors are rarely, if ever, addressed by the proponents of “new” solutions to peer review or scientific publication. These calls for change have been raised frequently recently. Of course “everyone” knows peer review is broken, a situation especially apparent when you’re at the receiving end of poor review practices. However, scientific reasoning is often tossed out in the pursuit of answers, the typical trap of problem solving without clearly understanding the problem system.
In the same (current) issue of The Scientist that argues for ending peer review as we know it, some scientists (usually in physical sciences) are now proposing that all scientific publication can be managed by the individual scientists. The solution is to post manuscripts and data on their “websites.” This is also proposed as a radical solution, but it is neither radical (it is an old idea) or a solution (the mess it will create is worse than the current system). Read in The Scientist Publish or Post?
A new European-funded initiative is advocating an entirely new system of science publishing, in which scientists avoid the hassles of traditional peer review by taking a quietly radical step: post their results on their websites.
They suggest making research — including formal manuscripts, datasets, presentation slides, and other presentations — available through the web without any sort of traditional peer-review process. That research would then be searchable and citable by the rest of the scientific community at no cost.
What I find consistently missing from such proposals are serious considerations of the richly intertwined social system that has formed around scholarly publishing. Activist scientists treat this issue as if it were a simple matter of problem solving. There is very little thought about the systemic impact of the proposal, based on understanding the underlying causes and drivers for the broken system. Yes, I am accusing these scientists of not having done research in this field, but making proposals based on their own authorship and reviewing experience. We call this first-person design, and sometimes it works, when the product is a “point solution” or an invention. But when the product is a social system, a larger networks of perspectives and stakeholders must be included.
There are so many questions this begs, that publishing systems have figured out, for better or worse. Publishing is not a simple matter of “using the Internet.” Your research and sites may be blocked, in accessible, poorly maintained, poorly ranked. Without social aggregation – not social networking, but a system you have control over – your work may be missed.
There are actually several great ideas here, ideas that my collaborators at U Toronto’s Laboratory for Collaborative Diagnostics have been working on for some time. Several projects and early research efforts underway have been using an interdisciplinary approach of social sciences, prototyping, and design research to make serious proposals for:
- Interpretive Collaborative Review – A novel review-generating system based on a multi-perspectival analysis of literature focused on a shared concern among a group of editorial reviewers.
- BioTIFF - A flat-file continuous storage system for recording and maintaining large image test data, data sets and personal health information, searchable and sustainable.
- Data-first publishing – Providing a means of publishing and indexing data for collaborative analysis, before reductive analysis and interpretation.
Not that we have the answers, and in fact, I’m writing to suggest that we do not have the answers yet. I just believe the answers being proposed are also quite insufficient. We should be investigating the social systems of publishing and review based on the purposes of publication, not the activity of publishing. That leads to underdetermined solutions.
What is the desired long term outcome for scientists, institution, the labs, and the research sponsor? Do scientists really want to take responsibility for maintaining their own websites in perpetuity? Is the individual scientist really the best unit of publication? What about multiple author situations? Who really owns the research, is it always the individual author? The trending interest in more special issues?
If scientists don’t have the time to peer review today, how are they supposed to maintain publication sites, and maintain standards and indexing? Will there be some support for this? Will they be trained or will it somehow evolve into consistency?
The biggest problem I see is not the means of publication but the exhaustion of readership. Publishing processes help focus quality, and reader attention is the most limited resource. The sheer volume of new publications today, nearly a million a year in the biomedical and life sciences, means that a very small proportion of papers are ever read, and an even smaller number are cited. Would individual authorship help this problem or make it worse? Just “publishing on the web” assumes that Google and some to-be-named social networking will solve the problems of indexing, findability, and even quality. I don’t see general purpose internet services as a reliable or responsible answer. We need to create better tools for qualifying publications for the human limitations of readership, and we must do this in partnership with all stakeholders, including libraries, publishers, and technology providers.
By Designdialogues, on August 14th, 2010
The 2.0 technology trends of new media, enhanced web applications, data-driven apps, and social media have advanced the sophistication and interaction of applications in most consumer domains. And co-occurring with this trend, the last three years have been filled with pronouncements of revolutionary changes in healthcare and personal health management envisioned by democratizing health information and enabling communications among people with very specific shared health concerns. Both technology innovations and consumer healthcare resources are included in the Health 2.0 arena.
An exemplary Health 2.0 trend is that of health seeking communities. Several early and progressive health community sites have taken hold (Patients Like Me, CureTogether) with adoption from people in various health “communities of concern.” There are numerous specialized disease and patient community sites in every niche and using every wiki and social service. While these are not intended to replace patient communication with health professionals, they serve complementary purposes, enabling conversations between people with similar concerns and questions. I don’t think we can say how these developments will impact healthcare practice innovation in 3-5 years, because but for now they are leading community resources for those with diagnosed and chronic and complex syndromes requiring creative and consistent self-care.
I start with a clear example since it appears that the conceptual models surrounding Health 2.0 share less agreement with each other than the Web 2.0. models (which were largely originated by Tim O’Reilly and developed from there.) And this should be seen as a healthy trend – early agreement around the meaning of innovation tends to limit its emergence and novelty. If current business and policy stakeholders take ownership of emerging concepts before the ideas have had a chance to organically develop at the level of user innovation, the future concept becomes colonized by those preserving their stake in the present.
How far do we really expect the web to take us? Health is not an information problem, certainly not solely, but a personal and social agreement acted on by commitments to future health. A systemic health revolution is not a function of better web applications, but of policy and practice changes. Information follows, it does not lead, these changes. Take a close look at the mess of large-scale electronic health records (EHR) systems if you need a reminder that information does not lead.
Susannah Fox, with the Pew Internet Project, posted a brief piece on Kevin MD titled “Will patients embrace Health 2.0?” She opens with a physician’s quote (below) that rings true, even if frustrating to those of us working on the next revolution. She reflects on responses from physicians that remind us that complex social systems are thick networks that require engagement on the ground, and that change may look more like community organizing than website design.
So we can all sit and perfect the tools for a few folks that never needed them anyway, or we can recognize that the kinds of solutions required for healthcare in the US today have nothing to do with fancy IT, or prioritization on search engines, and everything to do with low-tech, unsexy approaches toward grass-roots public health. Sorry to be the voice of reality guys.
You may start to rethink digital and product strategies that require health-seekers to make commitments based upon remote participation. Health and care are localized and personal. As Nicholas Christakis and Fowler recently illustrated (with analysis from the longitudinal Framingham Study) individuals create and sustain healthy and unhealthy networks. What you do locally makes a difference to everyone in your network. And these networks also take on a life of their own, as the field of public health reminds us that health and disease are both individual and distributed social phenomena. Connectedness appears centrally in the emergence and design of Health 2.0 resources.
Several models of Health 2.0 have been promoted, which present a wide range of opportunities for change and interaction. Dr. Scott Shreve created the compelling Virtuous Cycle model, illustrating factors for practice and policy innovation:
Notice that the patient is not fixated in the center of the model, and yet the patient-centered view is a significant trend in Health 2.0 overall. First of all, there are several different models and visions of Health 2.0, nicely listed and defined in Wikipedia.
What I find interesting about this model is how it shows the opportunities for change as independent of technology. There may be actual systemic leverage points in the model. Leverage points are the concept that an intervention can be planned and timed to have an accelerating effect on feedback to enhance a system of activity. In this model, many of the right features are identified: Incentives, Pricing, Service change, Practice change, and to some extent information. But the model presents a landscape of values more than it does technology. While these values are clearly drivers for change, they are not necessarily patient-centric. A systemic view of health leads to many other factors than the individual, whose immediate health may be of concern. I also think it oversimplifies the complexity of systemic healthcare to focus radically on the patient. Being a patient is a “state: that individuals pass through, it is not a personal identity (well, for most of us its not).
The value that I see in Shreve’s model is that of a systems cycle centered around a health condition, a cycle that prompts individual health-seeking and care, the intersection of which being the state where the “patient experience” shows up. Here we see, as Toronto’s collaborative informatics researcher Peter Pennefather says, “the disease is centre stage.” The emergence of illness creates the conditions for things to happen in care and practice.
So then I might not refer to this model Health 2.0, because it risks confusing the systemic innovation suggested in the model with the technology innovation universally implied by the 2.0 appellation. I see it as a systemic health innovation model, that integrates Health 2.0 technologies. That may be a limiting view – I’d like to know what you think … Apparently, according to Wikipedia, the possibilities for Health 2.0 are “unlimited.” That’s fine, but innovation requires that we reason about the systematic impacts of an intervention. We need to start somewhere, a point of leverage, that we think will make a difference.
Health 2.0 has a significant impact in changing practice, and these informed voices are critical accounts influencing the future field. The scope of concerns any doctor faces in practice, business, and patient decisionmaking is beyond the complexity those in other professions face. Engineers, architects, even lawyers have tools and processes that help them limit the scope and range of concerns in their practices. But physicians deal with both individual patient health problems, teaching or (if private) running a small business, and ever changing regulatory complexity. So when a new wave of technology change occurs in medicine, it should not be a surprise that they display a fairly conservative adoption cycle mediating their acceptance and utilization of new media and information resources. Therefore, there may be many flavors of Health 2.0 – and it may do a disservice to innovation to aggregate them into a category based on information technology, one that may not hold together well when developing the different vectors of healthcare design and innovation.
(Finally, a plug for Kevin MD’s blog - it presents a fantastic wealth of connections and ideas for the plugged-in physician and health researcher. I find it one of the best sources of links and linkages for scanning the horizon of emerging issues in the world of professional practice.)
By Designdialogues, on June 19th, 2010
As I’ve continued to develop material for the Design for Care project, I’m struck by the difference between design for practice and design for individual health-seeking. In designing for practice, ethnographic research and work domain analysis enable us to understand the range of activities and scope of work performed in professional work. A rigorous analysis of an activity system enables us to design services and information products that fit the work practice and cultures of use.
Professional work is highly consistent, at least within institutions. As with other high-hazard, complex skilled work, healthcare practice is regulated by law and professional societies.
While we can study medical and patient care practices in situ and on paper analysis, we will usually never design in the first person. We, design researchers, are not health professionals. And when we are, as many physicians by training are professional informatics specialists, we must separate our personal interaction needs from those of the designated practice being designed for.
Designers must always maintain a cognitive on-guard system to ensure we don’t “go native” and believe we have the perfect solution. Or, we may start to fixate on a given solution or pathway based on early concept successes or insufficient research. This is a well-known occupational hazard, and I have created different routines to ensure design and research are kept at a safe distance from each other. If I am leading research, I might have an associate perform the majority of design, giving us a dynamic collaboration. The client might allocate their designers to rendering new concepts, which research can critique. Or we exchange design leads between research cycles, work in parallel concepts, and so on.
But now consider designing for the health consumer. Assume that is you, and from time to time, all of us have compelling health seeking need situations. So what do we do when we are essentially designing for situations we are ourselves subject to? How do we maintain designerly distance when we are at health risk ourselves? Should we take action and build a new disease or condition-oriented community? Can we design experiences and services in the first person context?
Industrial designers are often the quintessential first-person designers. It may be one of the defining differences between complex systems design and the design of things. In ID we are drawn to design things we already love (cars, bicycles, kitchen tools) and we are taught to design for ourselves as primary users. We know what we want. In the world of things, there may be plenty of other choices for consumer. Often the selection between one car and another, with all common features considered, is style and grace. Online services may have fewer competitors, and to get the interaction wrong may cost the business model. And in health services, adopters can get harmed if the product incurs error situations or is puzzling to use in times of stressful use.
Consider people that live with diabetes. Many product designers are clearly first-person, or are close enough to the daily care requirements of diabetes to be a first-person designer in the situation. How do first person innovations come to market in the huge, regulated insurance or government payer healthcare system?
One way is that designers and health seekers take action. The market for diabetes innovations could be much better indeed. The recent Diabetes Mine innovation contest (for which I was a judge) just ended, with $21K in 3 first prizes given to three deserving contestants. IDEO will be helping them with formulating stronger design proposals and making their concepts market-ready. These winners were:
- Finn the Glucose Fish, a practical, ready-to-go design for a portable kid’s glucose monitoring kit. This one suggested the start for a simple system around which a platform or other products could be built (since not every kid will want a fish kit, forever!)
- Test Drive, a glucose reading-dependent vehicle interlock that requires the input of a good glucose measure to enable a diabetic driver to start the car (in this case, designed for parents with teenage diabetic). While this idea may not have a large market, may have some readiness issues (and I have concerns for its level of control) we believed this to have true social merit. The application could extend to commercial drivers, commuters, professional uses of vehicles and could lead to better insurance coverages for these drivers (or pilots).
- Zero, named perhaps for its perfect ring-like shape, the most innovative concept and perhaps the least ready for prime time as well. It was well conceived and envisioned, and sets a high bar for the vision that future products might aspire to.
Most of the finalists in the Diabetes Mine contest were clearly first-person design concepts. If I studied the diabetes health situation, and perhaps if I simulated the everyday experience, I might be better prepared to propose a meaningful design concept. It is not a matter of user-centered design. An experiential gap exists between authentic health seekers (patients) and non-patients. The non-patient will not have the awareness, the sensemaking of situations, and the motivation to design the breakthrough in a domain of personal healthcare. We do not have the authentic need to create “health hacks” that help us get through the day and live with our pain, regimes, schedules, discomforts and supports, care communities. The distance from health hacker to health designer is closer than we think.
By Designdialogues, on June 3rd, 2010
[110] in the Methods You Don’t Use Yet series
Expert Roundtable Review
Problem: For a product or service inquiry, we often see the need to rapidly gather highly relevant feedback and informed opinions on a new concept. A similar problem is noted when a project team is identifying the opportunities for innovation and must conduct a rapid but deep scan of the current products or competitors in the field. When the problem necessitates access to highly informed professionals, the focus group, field study, or survey methods are not feasible, at least within the timeframe necessary to make an initial business decision.
Solution: The Expert Roundtable Review, based on the Heuristic Evaluation [35] and Videoconference Focus Group [42], provides a dialogic tool for engaging a small group of informed participants online and by teleconference at a time convenient to their schedule.
Use When: Useful for collecting responses, opinions, and insights from busy professionals and respondents that may be difficult to recruit for focus groups, interviews, or even surveys. Also useful as a technique for convening an online focus group supported in advance by an evaluation or ideation activity explored within the focus group.
Use for: Product opportunity appraisal, Early concept design, Project planning and business case phase. Roundtables can be phased in sequence, starting with competitive analysis, then early concept testing, followed by prototype evaluation (with different samples recruited from the same population for each of these phased activities).
Process: The researcher identifies the need for the session based on the customer/user segments needed for the study. The participants can be recruited and scheduled from 10 – 14 days before the roundtable session. Incentives paid to participants are commensurate with position and expectations.
Typical steps involve:
- Identify candidate participants and their contact information, and circulate invitations with 2-3 date/time options. (If two sessions are needed, this is a way to determine the best allocation.)
- Prepare a draft evaluation worksheet for use in the homework, the advance task given to participants that drives the moderated discussion. Refine the worksheet to provide a document allowing participants to review and evaluate (typically) 2 of 3 or 4 options in some depth. These are often online websites or services that would be known to or of interest to the participants.
- Refine the heuristics in the worksheet (questions based on categorical criteria) so that each option is interrogated by 12-20 questions in 4-5 categories. Review and refine the draft worksheet with members of the service design team so that it reflects consensus on priorities.
- When candidates respond to preferred dates, confirm their appointments and provide the blank worksheet with instructions for advance preparation before the teleconference.
- Remind participants one day before the call and ask for worksheets in advance of the call.
- Moderate the teleconference with 4-7 participants (and with invited clients and observers), using the worksheet responses as a guide to the discussion. Record and transcribe the discussion.
- Prepare a summary report by compiling the worksheets into a composite review report, showing participant priorities in summaries based on the information collected. Some researcher commentary on the report helps contextualize the meaning for readers.
I have used this method dozens of times for situations where we convene experts in a domain to quickly review a variety of current resources to evaluate their fit to practice and to strategy. We often start with competitive analysis sessions, and as we learn more, build prototypes and review those head-to-head with the competition. After each set of sessions, a summary report is provided along with a discussion of the findings and value gained from the scan.
By Designdialogues, on May 27th, 2010
Consider design research – is it a discipline or no? Consider design researchers – researchers or are we really design consultants? A discipline has a body of knowledge, and a clear way of contributing to literature so that we know what we know. A real discipline has a theoretical base, and ways of using that theory toward outcomes in line with accepted values of the field.
In design research, what are the guiding theories and epistemologies that we recognize as credible and meaningful? What are the underlying philosophical assumptions that distinguish design research from market and user research? From social sciences or business research? Is it all really just (perhaps) a difference in methodology? And if so, isn’t that just technique and an activity that only supports the instrumental goals of a design project?
Even if design research is not leading innovation per se (see Don Norman’s Technology Leads and the recent DRC 2010) we should be leading with theories that help us make sense of what we’re inventing. If we are not creating contexts and joint understanding of impacts, then we are merely serving the instrumental purposes of products and clients. Sure, we can speak on behalf of our “users” but do we really have something to say ourselves? A recent (and ongoing) discussion on one of the old-timers design discussion groups surfaced this suggestion on the impact of theory in design:
“An effective theory is one whose purpose is clear and that defines and relates its elements in terms of the situations it addresses. It clearly communicates this structured knowledge and supports the actions necessary to realize goals regarding the circumstances it models. It provides evidence of its own effectiveness and produces useful knowledge.” (no citation yet)
Do we have one of these? Back before ethnography became methodologized into corporate practices, this was a critical and very real concern. Critical ethnographers (such as Bonnie Nardi) and ethnomethodologists (such as Mark Rouncefield) had the platform then. Theory was not drawn up based on fit to client, but to a great extent on temperament. On personal values, stance, perspective, and even aesthetics. What counts as “useful knowledge” is based on perspective and values, not on what a client says. We are the ones who have to make that transformation.
Back to why sensemaking matters as theory in experience design then. Sensemaking research has developed, without our help, over a 20-30 year period. It has increased in currency and applicability, where other venerable models of understanding (contextual inquiry, usability testing) have dropped away as primary research models. You can publish sensemaking research, while you really don”t want to publish usability research. Perhaps there is something to say here.
Brenda Dervin has positioned sense-making methodology in the literatures of communications and information sciences. Yet it is interdisciplinary work that builds a critical theory of individual agency interaction with structure. Unlike other critical theories (e.g., structuration Actor Network Theory) Dervin sense-making offers user experience a clarifying (if not actually unifying) theory of understanding human experience and behavior with respect to systems and structures.
User experience is surprisingly void of a theory of human experience and personhood. While we continually use methods informed by hermeneutic epistemologies: personas, empathic design, contextual inquiry - most of us in UX would not (really) know what hermeneutics means and why it matters. Let alone Dervin’s “quadruple heremenutic.”
The focus on method to the expense of understanding is a problem with interdisciplinary fields and of design in general. The current ways of thinking of user experience, and to an increasing extent, design thinking is almost entirely methodological. It is absent of what Dervin calls meta-theory, and tends to focus more and more on clever ways of engaging the user and adapting ethnography to design. Theories are tacked-on in UX, as post hoc working models of what we believe to be significant. We are reduced to explaining the monolithic social net-mare of Facebook in terms of usability or media consumption. We may be losing our ever-thinning opportunities to make a difference by chasing after fashion and phenomena. We are not yet redesigning the purposes of systemic design and methodology for the emancipatory contexts demanded in healthcare, citizenship, city planning, and other social contexts.
Setting information free, the call of the Open movement, is not the point. We live in an information hegemony, according to Dervin. The questions we know how to ask about free information are channeled by habit, culture, and our relationship to structures. Free information is already colonized by our own blindness.
We do not even see how, for example, open access journals replicate the publisher system, just with in-house players chasing after the same public goods. We need ways to intervene that disentangle us from the fog of power that envelopes our thinking in the first place. Can sense-making do this? well, it was designed to.
A Theory of Experience, or of User Experience?
Do the different theories of sensemaking matter, and if so, whose sensemaking and how to go? How does this make a difference with respect to practice and outcomes?
It appears to me that sensemaking sets a ground floor under experiences with information, artifacts, and complex services. ecessary to describe and explain the user experience (as its called) of interaction with situations and information. If Dervin would allow it, my proposal for the completion of the macro and micro levels of a UX theory of experience require the addition of two compatible reference models in a type of framework. The context of activity (or there is no “user” there) and micro-conscious enactment, as experienced within decisions and the sense-making moments of Dervin’s analysis. A larger frame of reference (e.g., macrocognition) may afford significant analytical power for a larger range of applications.
Finally, to offer a perspective on why this is important. We cannot address the tangles of power and powerlessness in our lives on an individual basis, which is where I may diverge from Dervin. Action research is a missing component of the sensemaking inquiry. If SMM researchers are to make sense of the “universal discontinuity of the human condition,” as Dervin suggests, we must also understand the response to discontinuity becomes a social act . People organize and socialize to remake their sense of concerns together. This may be what sensemaking misses today , and a possible position for eventual convergence between the different schools.
|
|
A Peter Jones Publication Blogs have evolved into several popular forms - mine is an old-school online publication of written works in progress. While the topics range widely, they reveal my interest in understanding the emerging social meaning of technology in use, finding better ways of designing for knowledge and organizational practices, and progressive interpretations of systemic innovation.
The title is meaningful - I see design processes as dialogic. Not just iterative, but design as languaging, both verbal & visual. We co-create & co-interpret in shared languages. A dialogic orientation requires we discover and appreciate the perspectives of all participants in a socio-technical system. Dialogue is performative designing - it requires both discipline and improvisation, to enable emergence of new meaning in human systems.
We hold these dialogues every 2nd Wednesday in Toronto:
Realize that dialogue has occurred when speaking leads to a new state of mutual understanding, and right action arises. This is also the purpose of designing.
|
