Friday, June 11, 2010

Restarting Interfaces

We recently changed five existing interfaces to "pass through" the Rhapsody interface engine that I am working with. These interfaces are made via a vpn tunnel between the two organizations.

That first day, one of the interfaces stopped working several times. I stopped and started the communication point on my end, and the connection was re-established and messages began to flow, again.

The next morning, one of the interfaces had not seen messages cross for seven hours. I again stopped and started the communication point, and messages began to flow.

I was told that "they do that all the time."

The next morning, three of the five new interfaces had stopped overnight. I restarted them, they connected and messages began to flow.

I have seen this sort of behavior with interfaces that cross through VPNs. When the interface is inactive, the vpn shuts down, but the receiving end still thinks that it has a connection. The sending end will try to re-establish a connection, but since the receiving side still thinks that it has a connection, the new connection is refused.

Being the cautious person that I am, I changed one of the three communications points to restart every hour. The interface would stop, reconnect, and messages would begin to flow, again. I wanted to see one of them work before I changed anything else.

Once I saw that this worked, I made similar changes to the other two interfaces that had exhibited similar issues. Two of the five original interfaces had not shown any issues. One was an ADT feed that constantly had traffic. The other received a "batch" of messages every morning. It would reconnect, deliver the messages, and then disconnect.

So far, the hourly re-start seems to be working. For years, these interfaces had failed. A trouble ticket would be opened, and an operator would manually stop and start the interface to get things working, again. Based on previous experience, I was able to diagnose the problem and implement a simple fix. This will save our business partner opening trouble tickets and our operators from manually restarting these interfaces.

One small step...

No comments:

Post a Comment