* What is Simple HL7 Message Search?

• Simple HL7 Message Search is a set of programs that allows quick searchs to retrieve any HL7 Ensemble messsages for immediate display and with links to the Ensemble message page by session ID for each retrieved message. All displays are in reverse chronological order.

• This search is different than an SQL search in that it uses a timeout feature to display the messages retrieved within a specified time period (default is 10 seconds) to avoid a timeout situation. The user chooses a drop down index name and a pattern (or blank) to search on. And the user can "next" indefinitely until all messages for that match are displayed. Also for faster searchs, you may want to click on the "1000" button instead of the "100000" button. But that will make no difference if there are fewer than 1000 messages for the index (like patient number). So a search starts at the most recent value for the date range and goes back in time up to 100000 if there are that many in the index or goes back as far as the "Back To" setting if less than 100000. Clicking on next simply takes the ending value in "Back To" as the "From" setting for a new search and adds 100000 to the ending "Back To" value. You could do the same thing by placing these values in the fields manually.

• 3 search fields for matching messages that have the specified patterns in the HL7 message.

• 3 exclude fields for excluding messages that have the specified patterns in the HL7 message or interface name.

• An interface filter to only include messages for a paticular interface. The default is all interfaces.

• A special index called "INTERFACE-". All messages are indexed by interface name as well as any indexes specified on the settings page.

• Optional display settings such as maximum number of segments, only display certain segments, only display a particular field, and status (completed, queued, suspended, etc.)

• Optional search settings such as case sensitive, all messages (show both source and target messages where applicable instead of source only), selectively dequeue messages, and bulk resend of all messages displayed (sent chronological order - oldest first per screen display - always set the criteria on the page to show all messages of interest - then do the search again with the send displayed messsages checkbox checked to actually send them). Hint-you can get more messages returned per search with a longer timeout than 10 seconds. 

Does the message search do anything else?

• Yes, it optionally outputs all messages for the date range in correct HL7 format to a user defined directory as specified on the settings web page.

• Output file name patterns are: Ensemble user name concatanated with the source name concatanated with the target name. If the messages found from the search are from different sources or targets, there will be several output files, with one for each source/target combination. The user needs to have read/write/create/overwrite rights to the directory. For unix systems, it is a good idea to have a special directory with all rights for users (chmod 777 /directoryname) such as /tmp. 

• All files remain and are appended to indefinitely until the user selects the delete output files button. Then all files beginning with the user name in that directory are deleted. Content is kept in a global so all files are overwritten the next time an output file is created with the old content in the global plus the new content. So manually deleting the files does not prevent old content from appearing in the files again - the delete output files button must be selected to delete the old content in the global.

• Messages can be resent and/or edited and resent to test systems (to resend an edited message 3 separate safety checkboxes must be checked as well as clicking on the send button). Edited messages that have been resent have a red notification on a search to show that they have been edited. The default target is always the original target. However any target can be selected for sending an original or edited message. 

What globals and classes are used for these programs?

• These programs use 1 global - "SimpleHL7Search" and approximately 25 classes with a package name of SimpleHL7Search. The reason for a large number of classes is so that the user can clearly see what is running in the Cache Processes Page (easy to understand what is going on).

• The programs use SQL queries of Ensemble messages by Message ID and Message Header ID, and to avoid the maxstring error, they read directly from the Ensemble segment global for HL7 message content.

• These programs never use the lock table. They safely and simply get content from globals when it is available. However, when using the DeQueue feature, the Ensemble code to remove items from the queue (##class(Ens.Queue).AbortItem) does appropriately use the lock table as designed by Intersystems. As a safety feature, this code is only run from a background job to avoid any possible problem with a terminated web browser session.

Does the indexing process and deleting of indexes use journaling?

• No.

• Every method in every class disables journaling as the first step. Journaling is not needed. If the system goes down in such a way that indexes are lost, simply reindex the last 1 or 2 days using the reindex feature.

• Delete and rebuild indexes whenever you like. It takes little processing, little disk space, and does no harm. The variable pause time setting (1 to 10 seconds) between indexing of up to 5000 messages at a time gives greater control of system resource utilization and leaves you in charge. If at any time there is a concern, indexing can be stopped with a click. If you need disk space, delete days of old indexes within seconds.

• So you have total flexibility to start, stop, delete indexes any time you like, with no measurable effect on your system.

Other indexing features (power users only)...

• Only index the messages you want. The settings page allows you to fine tune exactly which messages to index and which ones to exclude.

• Include or exclude based on message contents or interface name: Want to exclude messages for a particular interface? Just put that interface name in the "Does Not Contain" field. Want to exclude indexing of particular hospitals or locations - just put the HL7 field name in the "Does Not Contain" field. Or put the specific requirements that a message or interface name must contain in the "Must Contain" field. You can string together as many conditions as you like with the @ separator character. There is a special reserved word "VALUED". So you can put in the contain field something like: "PV1-3=VALUED@Interface1@PID-5.1=MOUSE" and only messages with source or target as "Interface1" with a location existing in PV1-3 and a last name of "MOUSE" would be indexed. No space would be wasted indexing other messages. This is just an example of what you can do. (NOTE: can't use PID-3.1~2 yet for these contain fields - only 2 levels for field and subfield at this time.) Another example - PV1-3=VALUED@Interface1@Mouse would index messages containing a location value where the words interface1 and mouse are contained somewhere in the HL7 message or the interface name. NOTE: the contain fields are not case sensitive.

• Include or exclude based on start or end dates: No need to index messages before or after date ranges of interest. Add an index for a day - just index on a particular field for a day or two to find the messages that interest you.

• Indexing by date and content is completely flexible.

Display of status information on the screens

• The Settings screen has the following display information.

  • Sum of most recently indexed HL7 Message IDs across all namespaces - automatically updates every second.

  • Sum of currently reindexed HL7 message IDs across all namespaces - automatically updates every second.

  • The date of the indexes being deleted - only displays when indexes are being deleted - each day is instantly deleted, but there there is a 1 second pause between each day's deletion so you can tell what is happening. Whenever, the number of days to keep the messages is decreased and there are messages to delete, this will display. Otherwise, after midnight the messages older than this number of days is deleted.

• The Search screen has the following information displayed whenever the screen is rereshed by any button other than the 1000 or 100000 buttons.

  • Messages not completely indexed is displayed in red when indexing is not caught up. This means you may not see the most recent messages for your searches.

  • The date and search information about the most recent background output file search. This information is updated every second while a search is in progress and shows date and status of the search including the number of messages that have been searched or written.

  • At the top of the messages returned in the search is display of the oldest message date checked for the current search going backward in time. If there are many messages to be displayed, you can keep clicking on "Next" and see older and older messages, which is reflected by this message date display. This date is the Ensemble message creation date adjusted to the current time zone. Also, the leftmost column shows how many messages backward in time for each message from the ending date selected.

Starting the message search for the first time

• When first running the settings screen, no indexing starts until you click on the Start Indexing button. . When you do click on start indexing, the background indexing job will start indexing the most recent 20 thousand messages (takes less than a minute) or the 1st HL7 message if there are less than 20 thousand messages. It you wish to index before that, you can use the reindexing feature to index up to 9999 days ago (or your first existing HL7 message if the date range is before the first existing message).

• There are up to 20 index settings at any one time (20 is arbitrary - if anyone wants more, just request it), with each index adding about 2% disk space usage per message indexed. You may delete indexes quickly at any time (deleting is fast - 1 second per day of messages - reindexing is much slower - about a million messages per hour).

• There are 8 default settings when the Settings screen is first run, 2 for patient ID, 1 for patient last name, 1 for account number, 1 for visit number, and 2 for order number, and a flexi-index for locations. You can overwrite the settings to anything you wish (click on the Item number), and use standard HL7 segment names, followed by a dash, followed by the field number, followed by a period, followed by a subfield number, followed by a tilde, followed by the repeat number. Like HL7 the latter subfield and repeat numbers are unnecessary if you want to index the contents of the complete field. Once the settings for the item are satisfactory, click on the update button. Examples follow.

• PID-3.1~1 for example gets the 3rd field of the PID segment. Then it gets the 1st subfield and then if there are multiples there it gets the 1st of the multiples. Then the message would get indexed according to the index name. If multiple items on this page have the same index, then the messages that match those criteria would all get indexed together under that index name. The default here is for the 1st patient ID and the 2nd patient ID to be indexed together with the index name "PATIENT-ID-" , but you may want them indexed separtately with different index names.

• So when your index settings are ready, click on the start indexing button on the settings page. At any time you can stop indexing by clicking on the stop indexing button. Then you can start any time by clicking again on the start button.

• If you don't like how your indexing is going, you can always delete all your indexes by keeping 0 days of indexes. However, after this finishes in a few minutes, you will want to set this number back to 30 days, or whatever you want to keep, otherwise when the automatic index delete process runs in the next few hours, it will delete all your indexes again. To reset the number of days both safety checkboxes need to be clicked on before clicking on the execute button. However, if you want to index before the current date and time, you'll need to set reindexing to the number of days you want to reindex. 

• When you leave the settings screen and go to the search screen (upper right button), you will want to set the number of hours your time zone is ahead of or behind Greenwich Mean Time. The default is -6 for Central Standard Time in the US.

Questions should be sent to ralphgcrawford@gmail.com.