The Email importing function accessible at System>Batch process menu>Data management> Data import/export>Email importing provides the ability to create new questions from emails arriving at specified email addresses, and create updates to existing questions from emails arriving at the RefTracker Service email address. In other words this function means RefTracker can assimilate new questions and question updates arriving by email at the email addresses specified here. When Email importing is set up in System>Batch process menu>Background processing, Email importing from the accounts that you set up here (in System>Batch process menu>Data Management>Data import/export>Email importing) will happen on a regular and automatic basis using the parameters (such as frequency of pickup) defined there. Your System Administrator (or any other email address/es defined in Parameter 1.3) is automatically advised if any email cannot be automatically processed so that they can manage these exceptions manually.
Despite being able to use this function to automatically accept questions sent by email, it is to your advantage to have clients use RefTracker forms to submit questions wherever possible. The reference interview provided by a RefTracker form ensures that you receive the information you need so that follow up interviews are minimised. When a client sends you an email they only send the information that they think is important resulting in a high percentage of follow up interviews being required. Further, when you receive an email, only minimal information can be collected for statistical purposes. For both of these reasons, when implementing RefTracker, you should be replacing all places that advertise your organisation's email address, with links to RefTracker forms where possible, in order to minimise the number of questions arriving by email.
Nevertheless, we recognise that many organisations, particularly high intensity ones like legal libraries, need to operate by picking up new requests from emails, and this email importing functionality provides a comprehensive solution for organisations that operate in that way.
Most RefTracker systems will utilise this email importing functionality to some extent, even if they only use the ability to automatically process update emails arriving back at the RefTracker service email address!
A common use of the new request import functionality is to accept new requests that come in as emails from other applications, like online databases.
Service email account and New request email accounts
RefTracker email importing handles two sorts of importing:
- "Service account" where the emails coming into this account will be replies and notifications in relation to the emails being sent out by RefTracker (it's the email address specified in your RefTracker parameter 1.2). Where an email comes into this account with a valid RefTracker question number in the subject line, a note containing the name, email address, subject line and body of that email,plus any attachments, will be added to the History of that question number, and the staff member currently allocated that question, if any, will be sent an email containing the text of the email and a link to the question in RefTracker so they can work on it. Where the email does not have a valid question number in the subject line, your System Administrator (or any other email address/es defined in Parameter 1.3) is automatically advised by email so that they can manage these exceptions manually (a one click option allowing any new requests arriving at this address to be resubmitted as a new request is included in that email) .
- "New Request" where the emails coming into this account will all be used to create new questions. For each email received into this account a new RefTracker question will be created that contains the subject line followed by the body of that email as the Question text, the sender's email address in the Client email field, their email name into the Client name field (if a name has been supplied), and any cc: addresses are saved into the RefTracker cc: field. Any attachments are added as question attachments. The new question is created using the Request form that you define in the set up, so any other data specified in that Request form such as Request type or Location will also be included in the question, and the processing specified by the form, such as sending a confirmation email, will be automatically undertaken. Your System Administrator (or any other email address/es defined in Parameter 1.3) is automatically advised if any email cannot be imported so that they can manage these exceptions manually (if the message requires action).
Sometimes emails that you want to become new questions in RefTracker, arrive into email accounts that are not being monitored by RefTracker such as an account being triaged by staff or even emails arriving into Staff personal email accounts. These emails can be forwarded to RefTracker as an attachment/s to an email to an email account that is being monitored by RefTracker. Click here and go to the section about "Forward as an attachment" for details of how to do this.
Setting up email importing for use
To set up the email accounts that can be monitored by RefTracker, go to System>Batch processes menu>Data import/Export>Email importing.
To use this function you will have to arrange with your email system manager to have POP3 or IMAP set up on the email address/es that you choose to use, or to have them set up as SMTP drop folders. If you have a hosted system you will need to use an email account that is valid for the hosted system. Altarama owns the reference-service.info, and libraryresearch.info domains so we can provide generic addresses for this purpose. If you have as secure hosted system, Altarama will set up the email importing accounts that use the domain used by your secure service.
Why use email importing
Unless your system is used for DeskStats only, you will want to use this Email importing function to monitor your RefTracker service email address (as defined by parameter 1.2) so that replies and non-delivery notices and other emails arriving at that address can be automatically added as notes to the question to which they belong. You will need to setup an entry here for your Service email address (and ensure that the Import email background task is set up for this account when you want it to start running automatically). Be sure that your Service email address is not being used for any other purpose than RefTracker outgoing emails.
Note that if you "forward" an email, the person who forwarded the email will become the client, instead of the actual client – so don't forward emails directly to these accounts unless you are prepared to manually change the client contact details. Instead use the RefTracker Import email by forwarded attachment process. See the section called "Emails forwarded as attachments to RefTracker email importing" in the How questions for follow up arrive in your system help page, for details of how this can be done manually or with just a few clicks using a Email client workflow function such as Outlook's Quick Step.
Safety of using email importing
RefTracker will import all of the email arriving at the email accounts that you designate to be monitored by RefTracker.
This means that spam arriving at those email account/s WILL be imported into RefTracker – the level of spam arriving will be determined by the effectiveness of the email spam filter in use at your organisation (or if your email address is provided by Altarama, by our up to date Barracuda spam filter with the latest filtering patches always applied). Not every spam email can be captured by spam email filters so you will definitely have some spam questions being created in RefTracker if you use email importing.
RefTracker can apply content filtering (parameter 0.20) that will result in questions created from spam emails being automatically marked as spam in RefTracker, but we have found that it will also identify real questions as spam. Our experience is that a human eye can easily identify spam questions displayed in the Open questions screen, and so the most efficient way to deal with any spam that gets through is to quickly delete the RefTracker questions created by spam emails.
Further it is possible for spam emails to get through your spam filter, yet still include code that might cause a security issue with applications like RefTracker. RefTracker addresses this risk through the following:
Handling HTML formatted emails
RefTracker can import all types of email - including text and HTML formatted emails. HTML formatted emails will be "simplified" before being imported into the RefTracker question field, to prevent any malicious code being picked up from HTML formatted emails. HTML formatted emails can contain formatting that is important to the understanding of the email, that could be removed by the simplification process, so we make the original source email available to be viewed from an [original email] at the end of the question summary.
Images embedded in the body of imported HTML format emails will be presented in RefTracker, in the position in which they occurred in the imported email, as a clickable thumbnail, if the image can be downloaded. If the image cannot be downloaded it will be represented by a clickable anchor tag. Parameter 0.60 provides details of how RefTracker handles images that cannot be downloaded by the email importing process, and provides security options for customers with RefTracker installed locally.
If you are using new question email importing you will need to have parameter 2.41 set to its default setting of "Dialogue only", or above, so that your clients can use the client interface with questions submitted as an imported HTML formatted email.
Setting up email accounts for importing
Each email address that you set up uses a Request form that you specify in the setup, to load the new question into RefTracker. Because different forms can be used for each different email address, emails arriving in the different email addresses can be handled differently according to what you specify in your form design. Allowing a different Request form to be associated with each incoming email address makes this system very powerful. So, for example, reference questions might come into one email address and result in the question being put in the pool. Be careful about setting a form used for email importing to send a confirmation email as this will mean confirmation emails will go to spammers - which is not desirable.
To add a new entry to the Email import setup click on the + sign on the top line of the table and complete the boxes presented. When you add a new account description here, an entry will automatically be created for it under System>Batch process menu>Background processing>Email importing. You will need to separately set the parameters for this batch process in order to start the automatic importing.
Account description: A textual description that identifies the purpose of this entry to you.
Email Server type: All addresses must be set up as "POP3", "IMAP" or "Mail drop folder". Choose "Mail drop folder" if your IT department has set up a physical folder where mail has been configured to deliver to a physical location by an SMTP server service provided by IIS6.0 (often called "SMTP drop folders").
Email server: The domain of the email server hosting the email account that will be used for this entry. If you are an Altarama hosted customer you will set this field to pop3reft.altarama.com.
Some organisations require that SSL be used when picking up email from a POP3/IMAP account. If you need to connect to a POP3/IMAP sever using SSL include the port number in this field. For example, for Gmail's IMAP service over SSL you would enter imap.gmail.com:993 in this Email server field, and for their POP3 service over SSL you would enter pop.gmail.com:995 in the Email server field. RefTracker's email importing service will automatically detect whether the email server supports SSL and use it if available. Otherwise it will default to trying to connect unencrypted.
Username: The full email account address that will be used for this entry. All email sent to this account will be picked up and brought into RefTracker.
Password: This field is not used for SMTP drop folders, BUT must be filled for data validation reasons, so enter any value here when setting up an SMTP drop folder.
Import Request form: This is the RefTracker form that will be used to create the new question from questions arriving as emails picked up from this account – so it is only applicable when the Email Account Task is "New Request". The form defines the labels that will be used to display this question and can be used to automatically include other information that is known because of the email address into which the question has arrived such as Request type, Client location, Priority (it will override any automatically detected priority), and Received via.
You can even set up mandatory fields in the form that cannot be completed by the email import, that will result in the form being marked as "incomplete" on import, which will ensure staff are taken to the Change screen to add the incomplete information (for example ensuring Matter numbers are added for questions arriving into legal libraries).
You can use a form that you use for other purposes, or you can design one specifically to suit the emails coming into this email address. See the "Hints for designing Request forms for use with email importing" section of this manual for important guidelines as to how to design a form for this purpose. We recommend that the Request form that you use is set up to NOT send a confirmation email, so that confirmation emails are not automatically sent to spammers on the occasions that spam email enters your system (which it is bound to do no matter how good the spam filters in use are!
In new systems a distribution Request form called "Email import" has been provided as an example of the form that would be used for this purpose (if you do not have it you can see it in our Playground system accessible from our supportsite). Note that even though this Request form is set with no client or staff availability, it works with the email import function as email import is a background is a function.
This distribution "Email import form" is also set up use special email templates that provide the client's Subject line in the Subject line of all correspondence re that request.
Dynamic lookups can be used with email importing to have client contact information other than the client's email automatically added to questions created from email imports - see the Dynamic Lookups manual for more details of this if you have the Dynamic Lookups Module.
Email Account Task: "Service account" defines that the emails coming into this account will all be replies and notifications in relation to the emails being sent out by RefTracker (in other words it's the email address specified in your RefTracker parameter 1.2). "New Request" defines that the emails coming into this account will all be used to create new questions.
Once an entry has been added to the Email import setup screen you can change it by clicking on the Edit symbol on the far right of its line.
You can delete it by clicking on the Delete symbol on the far right of its line.
Clicking the spyglass icon takes you to a function that allows you to test the setup of this account and see the import logs for that account. There is also a "Schedule email import process" button that allows you to do the import step of setting up the automatic import frequency. See the next section for details of how to use all these functions.
If you follow the instructions above you will have now set up your email account/s for importing. To utilise this setup you now need to set up the process for automatically importing the email arriving into those account using System>Batch process menu>Background processing>Import email, but you can test your setup and link to the batch process setup by clicking the appropriate spyglass "View email log" icon on this page.
View email log functions for an email account set up for importing
When your email account/s are set up you will want to test the setup, and then enable the Background process for them so that emails will be imported automatically. You may also want to view the logs that record the emails imported and processed by RefTracker.
To set up automatic uploading from the email addresses you have defined as enabled, or test your setup manually, and to view logs of pickups from this email account click on the "View batches. . . " spy glass icon on the end of the appropriate account setup line. This is the best place to go when looking for emails that have been imported for that queue (as opposed to the Background processing queue log tab) as it only shows occasions when there was activity, and it show emails imported when the import process is manually run.
The following screen print shows that the "Service email account" definition associated with the example firstname.lastname@example.org email address (the row with the grey highlight) has imported messages a number of times recently.
Date entered: the date and time at which the import process was run.
The [Show all messages] link shows a list of all of the email recently received into that account and the current status of each of those emails.
The [Show pending messages] link that appears for the Service email account only, and shows messages that have been imported but not yet processed (because they could not be matched to an existing RefTracker question). The email address specified by parameter 1.3 will be sent an email whenever a message is put into this Pending status. This email, and this screen provide [Send to <name> queue] links that allow each pending message to be resubmitted to a New question email account with a single click. The message you moved will disappear off this Pending list, and be imported by the email importing queue that you choose here, the next time that the email import process runs.
To set up automatic importing of emails from this account click on the "Schedule email import process" button. Details of how to use this Background processing function are provided in a later section.
To run a manual pickup click on the "Import emails now" button. You might do this in order to test your email setup (a "Failed when processing email account <name>"message indicates that your email account setup is incorrect – an email advising these failure details will be sent to the Active System administrator when this occurs. This ensures that the System Administrator is aware that the service has stopped operating when it is in live usage). The system will bring the details of all emails in that email account into a table of emails and then process each email into RefTracker creating a new question using the Request forms specified if the Mail Account Task is "New question", or creating an update to an existing question where a valid question number can be found in the subject line if the Email Account Task is "Service account". It may take some time to process the emails that have accumulated in your email account (about 1 second per email). After the upload completes a summary of the run will display below the "Import emails now" button as shown in the screen print above.
Your Active System administrator will be automatically notified via an email, whenever an imported email is rejected or left pending so they can apply the information manually if applicable.
View email log
If you want to view a detailed log of that Email import run you can click on the "click here" link provided in the summary, or, to view the log for any past run click on the spy glass icon at the right hand end of the line of the Account log for that run.Here's an example of a log for a Service Account run.
A [Show messages for all batches] link shows all emails recently handled by this process - great when you are trying to find a specific email.
The Question link – a link to the question that was created or updated (goes to the Answer screen, Answer tab for newly created questions and to the Details tab of the Summary screen for updates to existing questions). The System Administrator can use this link to work on this question immediately by clicking on it.
What happens when there is a problem with email importing
A number of safeguards and error logs are built into the RefTracker email importing process to ensure that the process operates smoothly:
If there is a problem with the process of email importing your System administrator, will receive an email detailing the error that is occurring.
Parameter 0.10 allows you to control the frequency with which these emails are sent when the issue is an email connectivity issue that is continuing to occur.
Despite these protections and limitations, email import infinite loops are still possible in instances where there is a problem with the allocated staff member’s email address – their email inbox is full, their email box has been deleted before the allocated staff member is Disabled or Deleted in RefTracker, or the email address is incorrectly entered for a new RefTracker staff member. It is also possible that an infinite loop could be generated by an email spammer.
Therefore the email importer also keeps a count of the number of “Update by email” notifications sent for any specific question. If the count exceeds a threshold (default 20 “Update by email” emails sent in relation to any one question within a 2 hour period) then an alert email like the example shown below, is sent to the system administrator. The notice is also copied to Altarama (cc'd to the To address recorded in the critical.xml file) as it is important that the issue be investigated as soon as possible by clicking the hyperlinked question number in that email – it takes you to the question’s History tab where the frequent updates will be able to be reviewed. The parameters for this functionality can only be set by Altarama using parameter 80.25.
The layout of the email sent will differ according to whether the email came into a Service email account or a New question email account.
The “Send to . . .” links resubmit this email for importing as a new question via the queue that you choose to click (there will be one link for every new question email importing queue defined in your system). When you click a "Sent to. . " link, the system will confirm that the question has been moved to the import queue (so that it can be processed next time that import queue is processed. If you are logged in, the message will appear at the top of the Home screen (displayed in a separate tab so it does not impact on anything you are currently doing and can simply be closed):
If you are not logged in the message will appear at the top of the login screen. You do not need to login. The message will display for 1 minute only - until the login screen next refreshes and the window/tab can be closed at any time:
The next time that queue is processed, the email that failed to import (complete with any attachments) will create a new question using the attributes of the queue that you chose to "Send to. . " it to. There will be a "Send to. . ." link for each new question email importing process that you have configured in your RefTracker system.
If an email arrives at the Service email address without a valid question number in the title, and there is only one open question that uses the email address from which this email came, the email sent to the parameter 1.3 email address will include extra information about the one open request as it might possibly be related. This is particularly helpful for things like mail box full notices and out of office notices that cannot be automatically handled by RefTracker using other techniques we apply. Here’s an example of the email that will be sent in this scenario.
If RefTracker only one question in the system that is from the same email address, a question number hyperlink will be provided that takes you to the Details tab of that question as it might be a match.
Use the new [Resubmit as update] function if the email is an update to an existing question.
Enter the Question number in the box provided and click “Check number”. A summary of that question number will appear. If it is the right question number, click “Update Question” and the email will be resubmitted for processing as an update to that question number, the next time the Service email address email importing runs
If you need to find the appropriate question number, use the [Search] link to have the Search function open in a separate window/tab. A search link is also provided in the "Unable to import" email.
A "Reject" link is also provided for scenarios where the email in not appropriate to be reprocessed (for example, it is a spam email). When you select the “Reject” option, the incoming email is deleted from the email importing queue.
Use the “View all pending email. . .” link if you have received a number of these emails – it will show you all emails that have not been able to be processed recently, with [Send to. . .] and [Reject] links allowing them to be reviewed and individually resubmitted from that list. Beware: using [Reject] completely deletes the email from the email import queue. It cannot be retrieved after [Reject] is used.
Don’t forget that you can use “Forward as an attachment” with $import (or the value set in paramter 0.14) in the subject line, to submit emails that arrive elsewhere (such as in your personal email), to RefTracker. Forward them to a new question email importing address for them to be imported as a new question, or to your Service email address for them to be imported as an update to a RefTracker question. If the email that you forward as an attachment to the Service email address, does not have a valid RefTracker question number in its subject line, use the “Resubmit” option, to allow the RefTracker question number that should be updated, to be provided, and the question updated by the next run of the Service email address email importing.
New questions created by email import
Where the Email importing function creates a new question from an imported email, the data that is collected by the Email import function from that email is applied using the Request form specified for this email importing address.
No matter what request form is used to import the email:
Here is an example of the log for an email that has been used to create a new question.
Here's how staff see it in the Open questions screen. Note that the Subject appears on a new line and bolded (just like it does in your email):
Where the Request form defined for use with this email account specifies it,
Viewing the History tab for that question will confirm that all the data in the email was imported into the new question, and will show which email importing address the email came in through. Clicking on "View email message" shows a copy of the email sent or received, with its header information (showing things like the Priority with which the email was sent).
Question updates added by email import
Once a question has been entered, clients can update the question using the links to that functionality in the email they received, but some of your users will use their email reply function to update their questions. If your system has email importing set up, any "reply" email that can be matched to an existing question will be automatically applied to that question as an update. Information about those that can't be matched will be advised to a nominated email address or the System administrator. Without email importing in place these replies will appear in an email address monitored by someone in your organisation.
When an email arrives at the RefTracker service email address, the system will attempt to match it to an existing question. It does this by:- searching the Subject line for a valid RefTracker question number- if none is found it will search the subject line of any emails that might be attached and matches on any valid RefTracker question number found there. This allows automatic matching of many types of non-delivery notices.- if a match is still unable to be made the system will search all open questions, and if only one open has a client email address that matches the email being processed it will be matched to that question. This allows for automatic matching of emails with simple email address typing errors.- if a match is still unable to be made, the contents of the email that can't be matched, and details of the sender, will be emailed to any email address specified in parameter 1.3, or if none is specified, to the System Administrator.
Once a match between arriving email and an existing question has been made, it is handled as follows:
If the email is from the client – where the client is defined as any email address in the client_email field (but not from any cc:, or bcc: fields):
If the email is from someone in a cc: or bcc: field:
If the email is from the allocated staff member, we can presume that the staff member knows they are updating the question and so they do not need to be advised:
If the email is from any other staff member:
If the email is from anyone else:
If the allocated staff member is Available, they will receive an email advising them of the update/reopen created by this imported email.
Client name: Client name obtained from the email (if any, otherwise "Client")
Also the System administrator we can click on the question number in the log and will be taken to the Details tab of the Summary screen for that question. In that screen you will see the note that was added to the question. If the System Administrator wants to work on the question they can click on Answer. In the Answer screen, the note just added will appear at the top of the History tab.
When you have finished examining the Email importing function and setting up the accounts that you will need for your organisation, go to System>Batch process menu>Data export/import>Time and cost Journal entry import.