Balancing precision and recall in the pharmacovigilance (PV) literature monitoring
What do precision and recall mean in lay terms?
In pattern recognition, information retrieval and classification (machine learning), precision and recall are performance metrics that apply to data retrieved from a collection, corpus or sample space.
Precision (also called positive predictive value) is the fraction of relevant instances among the retrieved instances, while recall (also known as sensitivity) is the fraction of relevant instances that were retrieved. Both precision and recall are therefore based on relevance.
In information retrieval, the instances are documents and the task is to return a set of relevant documents given a search term. Recall is the number of relevant documents retrieved by a search divided by the total number of existing relevant documents, while precision is the number of relevant documents retrieved by a search divided by the total number of documents retrieved by that search.
Why are literature searches required in pharmacovigilance (PV)?
In addition to monitoring the safety of the medicinal products through solicited and unsolicited adverse event reports, the marketing authorization holder has an obligation to review the worldwide experience with the medicinal product throughout the life cycle of the product—starting from the submission of the marketing authorization application to the revocation of marketing authorization approval. The worldwide experience includes published medical literature, considered to be a significant source of worldwide information on drug usage outside the controlled clinical study environment. This is achieved by performing worldwide searches in well-known medical databases, conducting more than one as a best practice.
Medical and scientific research databases are a collection of records relating to a set of publications associated with biomedical research. For any given record, each database has a structure that facilitates the organization of records and searching by various means, from simple text to complex indexing terms with associated subheadings.
Why precision and recall?
To interrogate a database, a search strategy/string is used, comprising more than a collection of terms. Search terms (text or indexed) can be linked using Boolean operators and proximity codes to combine concepts, increasing or decreasing the specificity of a search. In addition, limits to the output can be set.
When searching, the application of search terms means that the output should be less than the entire database of the records held. The success of a search can be measured according to precision and recall (also called sensitivity). Recall is the proportion of records retrieved (“hits”) when considering the total number of relevant records present in the database. Precision is the proportion of hits that are relevant when considering the number of records retrieved. In general, the higher recall searches will result in low precision.
What variables impact precision and recall of a PV literature search?
The two major components of PV literature search—the product term selection and the event terms selection (specific event of interest or generalized adverse event terms or special situation term)—are the variables that impact precision and recall.
To define the amount of anticipated work and the time frames in which it needs to be completed, a search needs to be optimized for output.
Thus, when performing a search related to an Individual Case Safety Report (ICSR), the aim should be to retrieve an output that is relevant to the company’s product(s) and the associated events in a given time frame. The output needs to be broad enough to cover all the articles that are relevant to fulfill the regulatory requirements; additionally, for expedited reporting in pharmacovigilance, the output should be specific enough, providing effective time utilization to complete the activities and ultimately perform regulatory submission within the timelines.
There is no acceptable loss of recall defined by any regulatory guidelines when searching published literature for pharmacovigilance. The use of search terms (free text or use of indexing) to construct more precise searches may assist in managing the output, which will allow for effective pharmacovigilance reporting.
PV literature monitoring: Steps to check the precision versus recall
Consider a scenario where setting up a search string for paracetamol for weekly ICSR searches is required.
Step 1: Select concept A (the drug name in this case)
Consider using all synonyms of the search concept A, as well as of concept B (discussed below), since some articles may refer to it one way while others another way.
These could be a different international nonproprietary name (INN) or generic names of the active, along with the brand names and clinical trial drug IDs, etc. For example, aspirin is also known as acetyl salicylic acid; similarly, acetaminophen or paracetamol-are same active ingredient, with different INN.
So add terms as below:
- Paracetamol
- Acetaminophen
Step 2: Select concept B (adverse events in this case)
Consider all synonyms of the search concept that need to be searched in reference to paracetamol.
For example, side effect can also be called toxicity or adverse effect, adverse drug effect, etc.
- Side effect
- Toxicity or toxicities
- Adverse effect or adverse drug effect
Step 3: Add a new term for each new criterion for search in concept B
Similarly for PV as well as side effect and its synonyms, we also need to search for special situation terms that are different concepts each, for example, overdose, occupational exposure, medication error, etc.
- Overdose
- Occupational exposure
- Medication error
Step 4: Consider combining all concept A terms with the “OR” function (Boolean operator) as well as combining all concept B terms with the OR function
However, ensure that concept A group terms are linked with concept B group terms with the “AND” function.
(Paracetamol OR Acetaminophen) AND (Side-effect OR toxicity OR adverse effect OR adverse drug effect)
Step 5: Add more limits to make the search more efficient but only if such limits are relevant to the purpose of your search
- Date limits: search period FROM DATE and TO DATE
As a best practice to be compliant with regulatory requirements, weekly searches can help with sufficient output and optimal coverage that are manageable in seven days, enabling submissions in 15 days
- Human limit (restricts output to human exposure of concept A and concept B)
For searches that have a broader concept or substantial data already published online like for PBRERs/PSURs of established drugs, we can use the following:
Step 6: Use truncation or a wildcard operator (this enables searching for different spellings of the same concept or for spaces)
- Sulfur or Sulphur: sul*ur
- Overdose or overdoses: overdos$
Using single-letter or multiple-letter/space truncation syntax (applicable to your search database) like ‘?’ or ‘*’ with similar spellings (Toxicit*) helps to cover multiple spellings of the same concept and broaden the search, increasing recall without loss of precision.
Step 7: Restrict searches to fields like only the title or abstract (this helps to search the concept only if present in the title or abstract)
- ‘heart attack’:ab,ti
This option retrieves records where the phrase “heart attack” is found in the title or abstract.
Step 8: Searches can be broadened (recall increased) without loss of precision by utilizing “indexed terms.” Some databases have hierarchical indexing (which is performed at the back end by the search database team) feature. This feature is similar to a thesaurus and links similar concepts under one category; however, these categories are not only specific to PV.
This can thus help you to search utilizing the Thesauri hierarchy structure available in some bibliographic databases like Embase or Pubmed and then search using broader or narrower (child) terms.
Step 9: Furthermore, to reduce the searches, without the loss of precision, we can use “text terms” with proximity operators (NEXT or NEAR)
For example, these are used to retrieve the words that must be within n words inclusive of each other in the same order as they appear in the search form.
- (adverse OR side OR undesirable) NEXT/2 (effect* OR reaction* OR event*