On the keeping of time

I told this story to a colleague this week, and it seemed worthy of a blog post.

Several years ago I worked for a large healthcare system. We supported more than 40 hospitals in seven states. I worked for the integration team and we developed and managed interfaces between all of their different IT systems.

They used an older mainframe based Patient Management System and a newer EMR. We picked up patient administration messages (ADT) and delivered them to the EHR and various other ancillary systems. We had an audit report that told us when a message took more than one minute to flow from the Patient Management System to the EHR, and we would have to explain why that happened. This was considered an SLA failure. One month we had a particularly high SLA failure report. As a junior developer I drew the short stick and had to investigate these failures.

I could see that we were receiving the ADTs and that they were going out to the EMR in a few seconds, well under the sixty seconds that we were permitted. I scratched my head and at the next status meeting I asked the innocent question "How are these times established." It was explained to me that these were the "system times" in the Patient Management System, the Interface Engine and the EHR. So I then asked the follow up question, "How do we know that the times are in sync?" What followed would be referred to in literature as a "pregnant pause."  Since I asked the question, I was tasked with finding the answer.

The Interface Engine and the EHR ran on unix boxes. These machines synced their time up with a time server at the US Naval Observatory every weekend. So, when I sat down with the EHR tech we verified that the times were in sync. When I sat down with the Patient Management System mainframe tech and we looked at the time in his system, it was a full 40 seconds off from the other two systems. Since that time "started the clock" for SLA purposes, this was an issue.

So I asked him, "How is the time set on your system?" The answer that I got was shocking.

"Well, when we do the quarterly Initial Program Load, whatever tech is performing the reboot looks at his watch and sets the system time to that."

This was a multi-billion dollar organization, and that is how they managed time. I was dumbfounded.

So, I asked, "How should we fix this?"

"Fix what?"

"How do we get the time in the mainframe to be set correctly?"

"The mainframe time is correct."

"Not according to the time servers at the US Naval Observatory."

"Yeah but....."

The mainframe was the center of the universe. The mainframe folks considered that the time in the mainframe was more correct than the US Naval Observatory.

Eventually we agreed upon a procedural fix which would require the tech that was setting the time during the quarterly reboot to bring up the web page of the US Naval Observatory and set the mainframe's time correctly.

My Experience at MHacks

MHacks is a hackathon at the University of Michigan. My company was a sponsor of the event this year, along with larger, more well known companies like Intel, Target and Disney.

I got to participate as a mentor, which basically means that project teams that are looking for technical guidance seek out mentors for assistance.

My specialty is Healthcare IT. There were two project teams that I had the pleasure of working with.

The first was a simple series of web baesd forms that asked some lifestyle questions and asked for the person's height and weight to calculate and display their body mass index (BMI). The actual calculation of BMI would be done in javascript on the web page. It has been over a decade since I did any web based programming, so I sent this team off to find some web based developers to get the syntax right.

The second team was more ambitious. They wanted to have web forms/apps that allowed a patient to update their demographic information and to schedule a visit with a doctor. I outlined the basic standards that support these data exchanges and then showed them the FHIR resources that were available to support this exchange. There are a couple of test servers out there that you can exchange with and they were going to use those as the back end for their app. I told them to emphasize that their solution was "standards based."

All in all, it was a very cool experience to see so many young people who are enthusiastic about technology.

It's all about how you recover

There is a lyric from Marillion that goes like this:

Failure isn't about falling down,
Failure is staying down.

I am in a leadership role in my present job. I work with a team of developers in the role of EDI Architect. We primarily move data between our internal systems and external trading partners.

One member of the team is a Senior Developer with lots of mainframe experience who is very new to the technology that we are using as our interface engine. This engine has a visually based development environment that can be challenging for a traditional "looking at code" developer to get comfortable with.

This week we promoted the first of his projects to production. The next day we realized that we had configured things wrong. He got very flustered. I told him to take a few deep breaths and then sit down with me while we figured it out. I told him that we needed to make sure that the change that we made was correct and that that was more important than making the change quickly.

We sat down and analyzed the problem and came up with a solution that we both agreed with. When we made this change, I brought each component up one at a time to make sure that they functioned as expected. We looked at the data flows and verified that they functioned correctly.

When we got done, I told him "We all make mistakes. It's all about how you recover."

I have now added a couple more things to the list of things that I look at when promoting to production. Hopefully, we both learned something.

My (almost) Annual Stanley Cup Predictions

Every year I look at the sixteen teams that are in the Stanley Cup playoffs and try to pick the winners.

I use a process of elimination. Single line teams or teams with suspect goaltending can be eliminated rather easily. This usually winnows the two conferences down to two or three teams. Then the fun starts.

So, here is my analysis by conference.

Starting with the Eastern Conference. We can expect that the Penguins, Red Wings and Islanders will be eliminated rather easily in the first round. None of these teams have the goaltending and balanced scoring that are required to make a long playoff run. That has Tampa Bay, the Rangers and Washington advancing to the second round. The Montreal and Ottawa series is interesting. The Canadians should advance, but I wouldn't put anything past Hammond. The Senators may be this years Cinderella team.

In the second round, I expect the Rangers to advance against the Capitals. The Tampa Bay and Montreal match up is a toss up. That would put the Rangers against either the Lightning or the Canadians. I expect the Rangers to advance to the Cup Finals from the East. They have the goaltending and experience to advance.

Now, for the Western Conference. The West is very strong this year. I am picking St. Louis, Nashville, Winnipeg and Calgary to advance out of the first round. The deciding criteria are, once again, goaltending, balanced scoring and experience. I have Nashville and Winnipeg advancing to the Conference Finals and Nashville advancing to the Cup Finals from the West.

That would give us the New York Rangers against the Nashville Predators in the Stanley Cup finals and I expect the Ranger's experience to prevail.

You can laugh at my predictions over the next two months.

Plans

My favorite quote about plans is from Helmut Von Moltke, who was Chief of the German General Staff during Bismark's era and defeated the French during the Franco-Prussian war, which led to the unification of Germany.

No Plan Survives Contact with the Enemy.

My variant on this is that "No Plan Survives Contact with Reality."

This maxim has served me well.

There is another interesting quote from von Moltke's nephew who was also named Helmut von Moltke. The younger served as Chief of the German General Staff at the beginning of the First World War. The Germans had an elaborate plan for going to war. It involved concentrating all of their forces against the French to quickly defeat them and then those forces would turn and deal with the Russians. This was an elaborate plan that involved railway schedules and was very German in its detail.

The Kaiser was convinced that he could keep the French out of the war and Germany would just have Russia to deal with. The Kaiser called von Moltke in for a meeting and told him that Germany did not need to go to war with France. Just turn the army around and we will go to war with Russia, instead. The response from von Moltke was:

Your Majesty, the movements of millions are not improvised. 

The German General Staff had a plan and they were going to stick to it. There was no contingency for going to war with Russia and not France. 

The Kaiser replied, "Your uncle would have given me a different answer."

von Moltke the younger was always being measured against his uncle. 

Anyway, Germany went to war with both France and Russia, just as they planned it. This plan involved crossing through Belgium, which meant that Britain joined the war. The rest is, as they say, history. 

Favorite Albums of 2012

I bought more albums in 2012 than I remember buying lately. I downloaded some and actually purchased discs of several. Here are my favorites that were released in 2012, in no particular order.

Genesis Revisited 2 -- Steve Hackett

This is a double disc set of mostly classic Genesis tunes that Steve has re-recorded. They stay mostly true to the originals. There are several of Steve's solo tunes from that era included.

Clockwork Angels -- Rush

It's hard to argue with Rush. These tunes are more rock and less prog, but I like them.

Sounds that Can't be Made -- Marillion

The band playes a bit more rock-like on this disc than on some recent efforts. H stil mumbles too much. I haven't bothered to look at the insert to see what he is supposed to be singing.

And The Band Played On - Music Played on the Titanic -- I Salonisti

This disc includes songs that were played on the Titanic's ill fated voyage. It's a nice way to hear the soundtrack of the doomed.

Masterworks -- California Guitar Trio

This is a nice collection of classical pieces given the CGT treatment. Very nice. The disc showcases the CGT's superb musicianship. Tony Levin plays upright bass and cello on three tracks.

Sea of Smiles -- Squackett

Steve Hacket and Chris Squire team up for some near prog. The combination is nice and works well. This disc does not rank up there with the best of their work with Genesis or Yes, but it is worth listening to.

Live Blood -- Peter Gabriel

These are versions of some of Peter's songs performed with an orchestra and not his usual band. There are some interesting interpretations of his songs on here.

Forbidden Planet -- Original Motion Picture Soundtrack

This is a re-release of the music from the classic film. I quite like this. I had forgotten how much the soundtrack added to the film.

All the Wars -- The Pineapple Thief

This is another nice collection of prog songs from The Pineapple Thief.

Sceneries -- Sylvan

More nice prog from Sylvan. Each song is presented in three or four parts. They range from mellow to rockin'.

Still More Stupid CCD Tricks

It has been a while since I posted one of these. Here are some mis-coded vital sign observations.

These developers do not understand the physical quantity data type. The bad values are highlighted. They also do not know how to validate a document against the cda schema, because this one fails.


<entry typeCode="DRIV">
<organizer classCode="CLUSTER" moodCode="EVN">
<templateId root="2.16.840.1.113883.10.20.1.35"/>
<templateId root="1.3.6.1.4.1.19376.1.5.3.1.4.13.1"/>
<templateId root="2.16.840.1.113883.10.20.1.32"/>
<id root="36e3e930-7b14-11db-9fe1-0800200c9a66"/>
<code code="46680005" codeSystem="2.16.840.1.113883.6.96" codeSystemName="SNOMED CT" displayName="Vital signs"/>
<statusCode code="completed"/>
<effectiveTime value="20120604"/>
<component>
<observation classCode="OBS" moodCode="EVN">
<templateId root="2.16.840.1.113883.10.20.1.31"/>
<templateId root="1.3.6.1.4.1.19376.1.5.3.1.4.13"/>
<templateId root="1.3.6.1.4.1.19376.1.5.3.1.4.13.2"/>
<templateId root="2.16.840.1.113883.3.88.11.83.14"/>
<id root="36e3e930-7b14-11db-9fe1-0800200c9a66"/>
<code code="8302-2" codeSystem="2.16.840.1.113883.6.1" codeSystemName="LOINC" displayName="Height"/>
<text>
<reference value="#vital0"/>
</text>
<statusCode code="completed"/>
<effectiveTime value="20120604"/>
<value unit="[in_us]" value="63inches" xsi:type="PQ"/>
</observation>
</component>
<component>
<observation classCode="OBS" moodCode="EVN">
<templateId root="2.16.840.1.113883.10.20.1.31"/>
<templateId root="1.3.6.1.4.1.19376.1.5.3.1.4.13"/>
<templateId root="1.3.6.1.4.1.19376.1.5.3.1.4.13.2"/>
<templateId root="2.16.840.1.113883.3.88.11.83.14"/>
<id root="36e3e930-7b14-11db-9fe1-0800200c9a66"/>
<code code="3141-9" codeSystem="2.16.840.1.113883.6.1" codeSystemName="LOINC" displayName="Weight"/>
<text>
<reference value="#vital4"/>
</text>
<statusCode code="completed"/>
<effectiveTime value="20120604"/>
<value unit="[lb_av]" value="154lbs" xsi:type="PQ"/>
</observation>
</component>
<component>
<observation classCode="OBS" moodCode="EVN">
<templateId root="2.16.840.1.113883.10.20.1.31"/>
<templateId root="1.3.6.1.4.1.19376.1.5.3.1.4.13"/>
<templateId root="1.3.6.1.4.1.19376.1.5.3.1.4.13.2"/>
<templateId root="2.16.840.1.113883.3.88.11.83.14"/>
<id root="36e3e930-7b14-11db-9fe1-0800200c9a66"/>
<code code="8310-5" codeSystem="2.16.840.1.113883.6.1" codeSystemName="LOINC" displayName="Temperature"/>
<text>
<reference value="#vital12"/>
</text>
<statusCode code="completed"/>
<effectiveTime value="20120604"/>
<value unit="[degF]" value="98" xsi:type="PQ"/>
</observation>
</component>
<component>
<observation classCode="OBS" moodCode="EVN">
<templateId root="2.16.840.1.113883.10.20.1.31"/>
<templateId root="1.3.6.1.4.1.19376.1.5.3.1.4.13"/>
<templateId root="1.3.6.1.4.1.19376.1.5.3.1.4.13.2"/>
<templateId root="2.16.840.1.113883.3.88.11.83.14"/>
<id root="36e3e930-7b14-11db-9fe1-0800200c9a66"/>
<code code="8480-6" codeSystem="2.16.840.1.113883.6.1" codeSystemName="LOINC" displayName="Blood Pressure Systolic"/>
<text>
<reference value="#vital16"/>
</text>
<statusCode code="completed"/>
<effectiveTime value="20120604"/>
<value unit="mm[Hg]" value="right 120" xsi:type="PQ"/>
</observation>
</component>
<component>
<observation classCode="OBS" moodCode="EVN">
<templateId root="2.16.840.1.113883.10.20.1.31"/>
<templateId root="1.3.6.1.4.1.19376.1.5.3.1.4.13"/>
<templateId root="1.3.6.1.4.1.19376.1.5.3.1.4.13.2"/>
<templateId root="2.16.840.1.113883.3.88.11.83.14"/>
<id root="36e3e930-7b14-11db-9fe1-0800200c9a66"/>
<code code="8462-4" codeSystem="2.16.840.1.113883.6.1" codeSystemName="LOINC" displayName="Blood Pressure Diastolic"/>
<text>
<reference value="#vital20"/>
</text>
<statusCode code="completed"/>
<effectiveTime value="20120604"/>
<value unit="mm[Hg]" value="80 and left 110" xsi:type="PQ"/>
</observation>
</component>