Rabu, 31 Oktober 2007

A Menagerie of Computers

I was recently asked what computers I personally run in the office and at home.

I exclusively use a Dell Optiplex D420 1.06 Ghz Core Solo with 1 Gig of RAM running Ubuntu Gutsy Gibbon Linux. The total weight of this 12" laptop with a 3 hour battery is 3lbs 4oz. Boot times average 40 seconds from power on to network connected. I run only open source software for all work that I do - OpenOffice 2.3, Firefox 2.008, and Evolution Email 2.12 connected to our corporate Exchange Server.

My wife (a graphic artist) uses a Macbook 13" 2Ghz Core Duo with 2 Gigs of RAM running OS X Leopard. The total weight of this laptop with a 2 hour battery is 5lb 2oz. Boot times average 35 seconds from power on to network connected. She runs a full suite of Adobe CS3 products.

My daughter (a high school student using Microsoft Office, instant messaging and multi-player role playing games) uses a Lenovo T61 2Ghz Core 2 Duo with 2 Gigs of RAM running Vista. The total weight of this laptop with a 2 hour battery is 5lbs 4oz. Boot times average 60 seconds from power on to Ctrl-Alt-Delete sign-on. Power on to network connected averages 2 minutes total with a clean Vista install. Although it's not a Gibbon or Leopard, Vista is certainly a resource consumptive beast.

Early this week, I installed Leopard on our Macbook. The ugrade from Tiger was automatic and effortless. One great feature is that Macs with Leopard can now search for files on Windows and Linux machines. From my wife's Macbook, I was able to search files on Ubuntu and Vista seamlessly using the Spotlight application. Our entire household is now sharing and searching files over our 802.11a wireless network. To enable Windows and Linux to search the Mac, I had to enable SMB Filesharing in Preferences, Sharing.

So what did I think of the new Apple Leopard OS X operating system?

As a CIO, I am most concerned about reliability, security, and interoperability. As a technology user, I care about simplicity, intuitiveness, and getting my work done efficiently. I evaluated Leopard wearing both hats.

Most users do not have time for training, so I explored Leopard's new features without the benefit of any instructional materials to evaluate their intuitiveness. My first exercise was to use Finder to explore PDFs, documents, spreadsheets and presentations via Cover Flow and Quick Look, two related document preview features that I think of as "sip" and "drink". Cover flows shows the first page of any document or multimedia file. Quick Look enables a detailed review of the entire file without having to launch an application. These worked well for all my file types.

I then moved on to the Spotlight feature to search files all computers in the household. Mac, Windows and Linux were instantly searchable.

Generally, I keep my Mac desktop uncluttered, using the Dock to manage applications and the Finder to locate needed files. The new Stacks feature enables me to create a "fan" or "grid" of my most commonly used files and automatically organizes my internet downloads. I found Stacks helpful in that fewer keystrokes are needed to retrieve commonly used files.

As part of a project to assess flexible work arrangements (see my blog entry on Flexible Work arrangements), I've recently been testing video teleconferencing, instant messaging, and groupware presentation tools. Since the new iChat features promised enhanced collaboration tools, I was eager to test drive the new remote desktop and document sharing capabiltiies. I found incredible clarity with the Mac to Mac videoteleconferencing tools which incorporate the H.264 video and AAC-LD audio codecs. Shared powerpoint presentations and shared desktops worked well over Google Talk and AOL's AIM.

Finally, I tested Time Machine, a backup and restore system similar to snapshot features I've used on IBM/Lenovo laptops. Having the ability to restore any deleted or reverse any accidental edit worked perfectly for me and will be very popular among users.

So, how would I rate these new features as a CIO? During all of my testing, Leopard booted, performed, suspended, restored, and shutdown without a hitch. Security of OS X is strong because of its Unix foundation. Interoperability of iChat and Spotlight across operating systems was excellent.

As a technology user, I found the new GUI features easy to navigate without training. As with previous releases of Mac OS X, the operating system hides the complexity of Unix in a way that enhances my ability to focus on work rather than the OS. My only complaint is one that I've voiced before: I want to install Leopard on non-Apple hardware, so that I can have OS X support on my 12" road warrior Dell subnotebook. In the meantime, I'll continue running Ubuntu Gutsy Gibbon, as it's the closest thing to Leopard that runs on a Dell. Steve Jobs - when's the subnotebook coming?

Selasa, 30 Oktober 2007

Personal Health Records

The exact definition of a personal health record (PHR) is still evolving, but personal health records hold the promise to make patients the stewards of their own medical data. PHRs may contain data from payer claims databases, clinician electronic health records, pharmacy dispensing records, commercial laboratory results and patient self-entered data. They may include decision support features, convenience functions such as appointment making/requesting referrals/medication refill workflow, and bill paying. However, most PHRs are not standards-based and few support an easy way to transport records among different PHR products.

The current landscape of PHRs includes four basic models:

Provider Hosted Patient Portal to the clinicians's Electronic Health Record - in this model, patients have access to provider record data from hospitals and clinics via a secure web portal connected to existing clinical information systems. Examples of this approach include MyChart at the Palo Alto Medical Clinic, Patientsite at Beth Israel Deaconess Medical Center, and MyChart at the Cleveland Clinic. The funding for provider-based PHRs is generally from the marketing department since PHRs are a powerful way to recruit and retain patients. Also, the Healthcare Quality Department may fund them to enhance patient safety since PHRs can support medication reconciliation workflows.

Payer Hosted Patient Portal to the payer claims database - in this model, patients have access to administrative claims data such as discharge diagnoses, reimbursed medications, and lab tests ordered. Few payer hosted systems contain actual lab data, but many payers are now working with labs to obtain this data. Additionally, American's Health Insurance Plans (AHIP) are working together to enable the transport of electronic claims data between payers when patients move between plans, enhancing continuity of care. The funding for payer-based PHRs is based on reducing total claims to the payer through enrollment of patients in disease management programs and enhancing coordination of care.

Employer Sponsored - in this model, employees can access their claims data and benefit information via a portal hosted by an independent outsourcing partner. An example of this is the collaborative effort of Pitney Bowes, Wal-Mart, Intel and others to offer Dossia, an open source application which enables patients to retrieve their own data. The funding for employer-based personal health records is based on reducing total healthcare costs to the employer through wellness and coordination of care. A healthy employee is a more productive employee.

Vendor hosted - several vendors are releasing products in 2007-2008 to serve as a secure container for patients to retrieve, store and manipulate their own health records. Microsoft's HealthVault includes uploading and storage of records as well as a health search engine. In a recent New York Times article, Google is reported to be offering similar features in late 2007. The business model for these PHRs is generally based on attracting more users to advertising-based web-sites, although the PHR itself may be ad free.

All of these models will be empowered by data standards for demographics, problem lists, medications, allergies, family history, the genome, labs, and text narrative. The Health Information Technology Standards Panel (HITSP) completed an initial set of interoperability specifications for demographics medications, allergies and advanced directives in 2006. In 2007, it completed problem lists, labs, and text narrative over networks and on physical media such as thumb drives and DVDs. In 2008, it will complete family history and the standards required to securely transmit genomic information. These standards will be used as part of Commission on Certification of Health Information Technology (CCHIT) certification criteria for electronic health records and personal health records over the next 3 years.

Another aspect of interoperability is interfacing home monitoring devices such as glucometers, scales, blood pressure cuffs and spirometers to personal health records. At present, most patients using these devices must manually type results into PHRs or call them into a provider because of the lack of uniform data standards in devices, EHRs, and PHRs. In 2008, HITSP will identify standards to ensure vital signs and glucose monitoring devices are interoperable. Continua is building a great foundation for this process by working with IEEE, HL7 and other SDOs to identify the most appropriate standards to support device interoperability and to identify gaps in current standards. HITSP and Continua will work collaboratively on standards selection in 2008 and beyond.

Privacy and Security are critical to health data exchanges between PHRs and EHRs. Privacy is the policy which protects confidentiality. Security is the technical means to ensure patient data is released to the right person, for the right reason, at the right time to protect confidentiality. The US currently lacks a uniform private policy for clinical data exchange. Local implementations are high variable and some organizations use opt-in consent, others use opt-out. The personal health record can help address this lack of policy. By placing the patient at the center of healthcare data exchange and empowering the patient to become the steward of their own data, patient confidentiality becomes the personal responsibility of every participating patient. Patients could retrieve their records, apply privacy controls, and then share their data as needed with just those who need to know. Since policies are local, the security standards built into PHRs need to be flexible enough to support significant heterogeneity.

HITSP has selected the security standards for the country which include audit trails, consent management, role-based access control, federated trust, and authentication. Personal Health Record vendors and device manufacturers will be empowered by these standards, which outline best practices for securing patient identified data transmitted between systems. Once patients trust the security of the network used to exchange data, adoption of personal health records and data exchange among payers and providers will marketly increase. Eventually, PHRs could also hold consent information, recorded via the HITSP Consent Management Interoperability Specfication, that can provide an easily queriable source for patient health information exchange preferences.

The evolution of today's paper-based, non-standardized, unstructured text medical record into a fully electronic, vocabulary controlled, structured interoperable document shared among patients and providers will be a journey. Standards are key and recent work in this area now provides the foundation for personal health records. Security technology exists today that is good enough. Early experiences with PHRs demonstrate high patient satisfaction, reduced phone volume to provider offices and less litigation by patients sharing medical decisionmaking with their clinicians. The time to implement PHRs is now and the only barriers are organizational and political, not technological.

A Green Approach to Storage

Recently, several folks from the press have asked me about "green" approaches to data storage, since we have 200 Terabytes (that's 2,000,000,000,000 bytes) of medical records online. Over the past year, we've begun the journey to reduce the power consumption of storing data.

1. Higher capacity drives. Just a few years ago, our drives were a few dozen gigabytes each and we had to keep a large number of drives powered up in our data center. Now, higher capacity drives (750 Gigabytes each) means fewer devices consuming less power. We also use Serial Advanced Technology Attachment (SATA) drives, which rotate slower and consume less power. Many folks were worried about the performance of these slower rotating drives. We did a pilot and put half our users on the SATA drives and half on Fiber Channel. Then we moved everyone to SATA. No one noticed the change.

2. Reducing the amount of data stored. My needs for storage are growing at 25% per year and despite all my attempts to reduce demand (i.e. deleting all MP3 files on the network every data), demand is hard to control. Recently, we've used de-duplication techniques to help with this. Our email system keeps only copy of an attachment sent to our 5000 employees, not 5000 copies. Our backup systems deduplicate files, so only one copy is stored. We've seen a 50% reduction in the space needed for archiving because of this, reducing the amount of storage devices and the electricity needed to power them.

3. Spin-down and slow-down technologies - turning off unused drives. Spin-down is controversial. Some believe the benefits are over-stated because periodic wake-ups of the disk for integrity checks, etc may consume more energy and shorten disk life cycles. Many vendors seem to favor slow-down technologies. They especially see this for backup media such as Virtual Tape Libraries (disk emulating tape). Long term, there is a prediction solid state drive (Flash RAM drives) costs will drop and permit more frequent use. These are more energy efficient and have no moving parts, making them easier to manage from an energy perspective.

Minggu, 28 Oktober 2007

Kill a Watt!

As a followup to my Friday post, here are a few more details of our "kill-a-watt" project. This is a multi-year effort by the staff to "green-up" the data center and IT electrical footprint that extends to the medical center.

1. Eliminate old monitors - Conversion from cathode ray tube-based (CRT) to LCD flat panel computer monitors. The typical power consumption for a CRT is .41 kwh's per day. A year ago, we had 7,500 CRT's. Today, we have reduced the number to 2,400 having replaced the others with 15" or 17" LCD flat panel monitors. The latter consume an average of .29 kwh's per day (17") or .11 kwh's per day (15"). Based on the effort to date, we have lowered the annual electrical power consumption for computer monitors from 1,122,375 kwh's per year to approximately 459,189 kwh's per year. This includes the impact of 500 additional monitors that were new to the environment.

2. Server Consolidation - Many of our computer servers in the data center support only one application. This is often requested by the application vendor. Virtualization software such as VMWare is now available that allows multiple instances of a computer operating system to exist on the same server. Each instance can be configured in a way that assures needed server resources (memory, CPU, etc) are available. We have retired over 20 servers in the past year through this technique.

3. Computer Center Utilities - There have been several initiatives related to the data center. Prior to 2007, we were tracking to exceed 200KW's of peak power consumption in the data center by 2010 based on the growth of electrical demand. The latter was caused by new storage, servers and switches. We were also seeing a 1:1 ratio between the data center load (servers, storage, switches) and the mechanical systems required to cool the data center and light it. In other words, for every watt used in the data center, we needed another watt to cool the heat produced.

Our goal was to stay well below 200KW peak load in the data center and reduce the 1:1 ratio. While we are not done, we have essentially achieved our goal. Today, we run at about 160KW at peak load; below the 170KW~ peak loads we have seen in the prior year. This is despite the increased number of terabytes of storage and applications the medical center has demanded we support. This has come about by consolidating servers, replacing older, less energy efficient devices, and reconfiguring equipment.

The data center load to mechanical (HVAC) load has also improved. Instead of 1:1, we are now running at about 1 watt of data center use for every .7 watt of mechanical load. This has been the result of better thermal management (e.g. reducing underfloor cables, using perforated inserts, shutting down an unneeded A/C unit, adjusting humidistats to increase efficiency, using blanking panels and other methods to better direct air-flow (i.e. hot aisle/cold aisle enforcement), and other techniques.

Kamis, 25 Oktober 2007

Some Like it Hot

As a Prius-driving vegan, I'm doing everything I can to reduce my carbon impact on the planet. This also includes an effort to build "green" data centers. My next few posts will be about the power consumed by the technology we use in healthcare. It's estimated that between 1.5 and 3.5% of all power generated in the US is now used by computers.

I recently began a project to consolidate two data centers. We had enough rack space, enough network drops, and enough power connections, so the consolidation looked like a great idea way to reduce operating costs. All looked good until we looked at the power and cooling requirements of our computing clusters and new racks of blade servers. For a mere $400,000 we could run new power wiring from electrical crypts to the data center. However, the backup generators would not be able to sustain the consolidated data center in the event of a total power loss. So, we could install a new $1 million dollar backup generator. Problem solved? The heat generated by all this power consumption would rapidly exhaust the cooling system, driving temperatures up 10 degrees. We investigated floor tile mounted cooling, portable cooling units, and even rack mounted cooling systems. All of these take space, consume power and add weight. At the end of the planning exercise, we found that the resulting new data center cost per square foot would exceed the cost of operating two less densely packed data centers. We looked at commercial data hosting options and ran into the same issue. Power limits per rack meant half full racks and twice as much square footage to lease, increasing our operating costs.

At my CareGroup data center, we recently completed a long term planning exercise for our unused square footage. Over the past few years, we've met increasing customer demand by adding new servers and power has not been a rate limiting step. However, as we retire mainframe, mini and RISC computing technologies and replace them with Intel/AMD-based blades, the heat generated will exceed our cooling capacity long before real estate and power are exhausted.

The recent rise in the cost of energy has also highlighted that unchecked growth in the number of servers is not economically sustainable. In general, IT organizations have a tendency to add more capacity rather than take on the more difficult task of controlling demand, contributing to growth in power consumption.

Power consumption and heat is increasing to the point that data centers cannot sustain the number of servers that the real estate can accommodate. The solution is to deploy servers much more strategically. We’ve started a new “Kill-a-watt” program and are now balancing our efforts between supply and demand. We are more conservative about adding dedicated servers for every new application, challenging vendor requirements when dedicated servers are requested, examining the efficiency of power supplies, and performing energy efficiency checks on the mechanical/electrical systems supporting the data center.

We have also begun the extensive use of VMware, Xen and other virtualization techniques. This means that we can host farms of Intel/AMD blades running Windows or Linux, deploying CPU capacity on demand without adding new hardware. We're connecting two geographically distant data centers together using low cost dark fiber and building "clouds" of server capacity. We create, move and load balance virtual servers without interrupting applications.

Managing a data center is no longer simply a facilities or real estate task. We've hired a full time power engineer to manage the life cycle of our data center, network closets and disaster recovery facilities. New blade technologies, Linux clusters, and virtualization are great for on demand computing, but power and cooling are the new infrastructure challenge of the CIO.

Rabu, 24 Oktober 2007

The Open Source desktop has finally arrived

Over the past year, I've run Windows Vista, Mac OS X (Tiger), and several flavors of Linux (Ubuntu, Debian, SUSE, Fedora, Red Hat Enterprise). Although it is true that Mac OS X is my favorite operating system for multimedia support, user experience and stability, it only runs on Apple hardware. As a road warrior CIO, I need a 12" subnotebook weighing 2 pounds, which does not yet exist in Apple's product line.

For the past 6 months, I've been running Ubuntu Feisty Fawn Linux with Open Office, Evolution email, and Firefox on a Dell D420 subnotebook. It has been good enough for all my computing needs, providing secure, reliable, easy to use software with a low total cost of ownership. However, there have been a few annoyances. Suspend/resume works about 80% of the time. When it does not work, my matchstick mouse locks up and I have to close the lid, resume again and all is well. Occasionally, upon waking from suspend, my laptop does not reconnect to my home wireless network. The Evolution email client can be slow to synchronize with Microsoft Exchange because it does not cache email locally and thus downloads all email headers from scratch whenever the email application is started. However, all office productivity, browser, and multimedia applications work flawlessly.

Tonight, as part of the normal Ubuntu updating process, I clicked on Update Manager to automatically replace the entire Linux operating system on my laptop with the next release of Ubuntu, Gutsy Gibbon. The new operating system downloaded, automatically installed itself and resolved every problem I have ever had with Linux. Congrats to the folks at Canonical who maintain Ubuntu and to the folks at Novell who have significantly upgraded their Evolution email client to meet the needs of Microsoft Exchange users.

I am completely confident in saying that "Linux your grandmother could use" has now arrived. With Ubuntu Gutsy Gorilla, everyone can have a free desktop/laptop operating system with all the productivity tools necessary to get your work done.

Two years ago, when I had dinner with Steve Ballmer, I explained that healthcare needs highly reliable, lower cost, more secure desktop software. He countered that new features are the highest priority of Microsoft customers and that it would be impossible to create a lightweight, reliable, low cost, secure version of Microsoft operating systems and applications given the demand for ever increasing features.

I am a realist and recognize that Microsoft provides many enterprise software products that will continue to have a strong presence in healthcare. However, now that Ubuntu is good enough, I expect that more people will try it and experience the advantages of running software with just the right balance of features, speed, and stability. And it's always free.

To try it yourself, go to http://www.ubuntu.com/getubuntu

Let me know how it goes!

Senin, 22 Oktober 2007

Exploring Instant Messaging

Per yesterday's post, over the next few months I'll be piloting the policies, technology and governance of flexible work arrangements. I live by Blackberry email, cell phone, web, and remote data access via SSLVPN. To expand my communications horizons, I'll be testing Instant Messaging, various forms of video teleconferencing, blogs, wikis, collaboration tools and group authoring tools.

Here's my summary of the Instant Messaging experience to date. As an email guy, it has taken some getting used to. I've done IM via AOL's Instant Messaging (AIM), MSN Messanger, Yahoo, Google Talk, a local Jabber server at BIDMC, and Skype's chat features.

To me, effective chat needs to work across all platforms, so I tested all of these platforms with my Ubunu Feisty Fawn Linux laptop, my Macbook, and my Dell Optiplex 745 desktop running Windows XP. For Linux, I used Pidgin and Gajim open source instant messaging programs. For the Mac I used iChat (AIM and Jabber) plus downloaded clients from MSN, Yahooa and Skype and for the Windows machine I downloaded clients from AOL, MSN, Yahoo, and Skype. For Google Talk, I also tried the Google web client that's part of Gmail by using a Firefox browser on all three computers.

My first impression is that IM can be an effective communication tool for realtime emergent situations, for quick questions (when is the meeting?), and for brainstorming as a group. One frustration is that my collaborators have accounts on different IM platforms. With email, I simply send to the address of each server used by my collaborators. With IM, I must login to the same service each is using. My Linux clients enable me to login to multiple IM services simultaneously, but I still need to create and remember the credentials to accounts on all these services.

Some of these services support video and voice chat. Here's what I found

AIM - Windows AIM client supports chat/voice/video, Mac iChat supports chat/voice/video, Linux Pigdin/Gaim supports chat only. All use the proprietary OSCAR protocol.

MSN - Windows Messaging client supports chat/voice/video, Mac aMSN open source application supports chat/video, Linux aMSN open source applications supports chat/video. All use the proprietary MSNP protocol

Google talk - Hosted implementation of the industry standard Extensible Messaging and Presence Protocol (XMPP) protocol and the extended Jingle protocol. Works with any standards-based chat client such as Trillian for windows, iChat for Mac, Pidgin/Gajim for Linux. Supports audio via a downloadable Google talk client for windows, iChat for the Mac. No audio support for Linux.

Yahoo - Windows Yahoo client supports chat/voice/video, Mac Yahoo client supports chat/video, Linux Pidgin/Gaim supports chat only. All use the proprietary Yahoo! Messenger Protocol

Skype - Windows Skype client supports chat/voice/video, Mac Skype client supports chat/voice/video. Linux client supports chat and voice only. All use the proprietary Skype protocol.

Bottom line - All provide text chat on Linux, Macs and Windows. Skype provides voice on all these platforms. None of these services provide video and voice on all platforms.

My conclusion is that text works very well as long as your collaborators are on the same IM service. Voice is problematic across platforms and has very uneven quality that's a function of many bandwidth bottlenecks from desktop to desktop via the internet and the IM service provider. Combined Voice and Video across platforms is not yet possible with IM.

Has anyone had a different experience?

More to follow as the exploration continues.