As interest in well-being increases and health care in daily life becomes important to lead a healthy life, dietary supplements market is also increasing globally.¹⁾ Since the emergence of coronavirus disease (COVID-19), increased awareness of health and the spread of self-medication trends are expected to accelerate the growth of the dietary supplements market and further expand it.^2-3) The size of the dietary supplements market in Republic of Korea has also continued to grow at an annual mean growth rate of 12% since 2016.⁴⁾

The definition, scope, and regulations on dietary supplements vary by country. Dietary supplements are defined by the US Food and Drug Administration as products taken by mouth that contain dietary ingredient, which are vitamins, minerals, amino acids, and herbs or botanicals, as well as other substances that can be used to supplement the diet.⁵⁾ They are also referred to as “health functional foods”,⁶⁾ “food supplements”,⁷⁾ “natural health products”,⁸⁾ “food with health claims”,⁹⁾ among many others. While dietary supplements can help supplement the diet, obtain vital substances the body needs to function, or help reduce the risk of disease, it should be cautioned that they are not intended, and therefore should not be marketed as to treat, diagnose, prevent or cure diseases.¹⁰⁾

While dietary supplements are perceived to be safe by consumers as they are “natural” or “herbal”, functional ingredients in dietary supplements that provide physiological or pharmacological effects can result in adverse effects or drug-drug interactions, and this risk can be further enhanced in special populations, such as children, pregnant women, patients with impaired organ function or immune function.^11-15) Although dietary supplements are available for purchase under the counseling of pharmacists at local pharmacies, as dietary supplements are readily available for purchase in stores or the internet without much restrictions, in most cases, consumers obtain information from various sources and make selections according to their perceived needs.

Along with the expansion of the dietary supplements market, the mobile health (mHealth) market is also growing at a significant rate.¹⁶⁾ mHealth refers to devices and behaviors for health management using mobile devices such as smartphones. In particular, in an era when facing each other has become difficult due to COVID-19, health management through mobile applications (apps) is receiving more attention and is being utilized more frequently among patients and the general public.^17,18) mHealth apps can be broadly classified into two categories: (a) medical apps, which are used for purposes of prevention, diagnosis, or treatment of diseases, and (b) nonmedical apps, which are those relevant to lifestyle, fitness, and well-being.¹⁹⁾ mHealth apps for dietary supplements can be a convenient way to obtain information on dietary supplements’ health claims, compare ingredients of various products, compare reviews, obtain personalized recommendations, as well as to manage medications and intake of dietary supplements. Such apps on dietary supplements would mostly fall into the category of nonmedical apps.

However, the quality of the mHealth apps is not guaranteed. In the app market, there are few restrictions on the app development, and as the development process is not complicated and the development cost is relatively low, the market entry barrier is rather low.²⁰⁾ While several countries have developed regulations for high-risk medical apps, most of the nonmedical apps are not subject to such requirements.¹⁹⁾ Therefore, it is currently up to the individual app users to appraise and properly identify apps that provide reliable and high-quality information and function. Since apps for dietary supplements can be directly related to the health and well-being of the app users, the need to ensure their quality and accuracy of the information and functions provided is even greater. In the meanwhile, a considerable number of apps on dietary supplements have been developed and are readily available to consumers without formal evaluation or peer review.

Therefore, this study aimed to systematically review dietary supplement apps released in English and Korean on the Google Play Store and the Apple App Store to examine major contents and conduct a quality evaluation of the currently available apps on dietary supplements using the Mobile App Rating Scale (MARS). Furthermore, we also aimed to evaluate the factors related to the quality of the apps and to explore the directions for enhancement of dietary supplement apps.

Systematic Search Strategy

This study was conducted by referring to previous studies that designed a systematic review of the mHealth application program by applying the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) tool for systematic review and meta-analysis to the app database.^21,22) Mobile phone apps on dietary supplements were systematically searched on Google Play Store and Apple App Store. Google Play Store search was conducted on a computer after all Google accounts have been logged off to eliminate the possibility of the results being affected by the account. Apple App Store search was conducted on iPhones. The following search terms were used in English and in Korean: “dietary supplement”, “health functional food”, “nutritional supplement”, “vitamin”, “natural medicine”, “natural products”, “herb”. The search was conducted between July 19, 2021 and August 05, 2021 by two researchers.

App Selection

The inclusion and exclusion criteria of the apps were defined a priori and are summarized in Table 1. In brief, apps that were searchable under the aforementioned search terms, those available in English or Korean language, and those with last updated date within 2 years of the search were included. On the other hand, apps aimed only at dietary supplement product sales, not relevant to dietary supplements, that do not provide information on dietary supplements (i.e. information on herbal plants only), with information only for non-human (i.e. animal) use, developed only for medication administration reminder function, intended only for healthcare professional use, or requiring subscriptions outside of the app for use were some of the main exclusion criteria.

Table 1

App selection criteria

Inclusion criteria

1. App retrieved on Google Play Store or iPhone App Store by the following search terms: dietary supplement, health functional food,

nutritional supplement, vitamin, natural medicine, natural products, herb

2. App available in English or Korean language

3. App that is last updated (or launched) after July 2019 (within 2 years of the search date)

Exclusion criteria

1. App aimed only at product sales (i.e. App that includes ~ Store, ~ Shop, ~ Shopping, ~ Mall, ~ Market, etc in the app title)

2. App that is not relevant to dietary supplements, does not provide information on dietary supplements (i.e. information on herbal plants only),

or only provides information on non-human use (i.e. animals)

3. App that simply serves as medication administration reminders

4. App that does not run or is not downloadable after three separate attempts

5. App that includes dietary supplement information at a minimum and/or information other than dietary supplement is mainly provided

6. App available in languages other than English or Korean

7. App intended only for healthcare professionals and not for consumers or the general public

8. App with no update record or that is not updated (or launched) within 2 years

9. App that requires subscriptions outside the app to use the content

10. App with less than 100 or unknown installation counts

11. App with no user star ratings

After the full names of the apps have been collected through the systematic search using Excel 2016, duplicate apps were removed. Then, two researchers independently screened through all the apps and excluded apps that did not meet the inclusion and exclusion criteria based on the title and description of the app on Google Play Store or Apple App Store. Any apps that were not searchable through the App Stores were also excluded. Any disagreements on the inclusion of the apps after the initial selection process were evaluated and determined by a third researcher. Apps that were included after the initial selection process were installed on either Samsung Galaxy Note 10 for Android-based apps or iPhone X and iPhone SE2 for iOS-based apps. Two independent researchers evaluated the installed app for the inclusion and exclusion criteria. If agreement was not reached between the two researchers, a third researcher independently evaluated the installed app and three researchers reached a consensus through discussion. Through the discussion, additional exclusion criteria were later added, which were to exclude apps with less than 100 or unknown installation counts, and those with no user ratings (exclusion criteria 10 and 11 on Table 1), as such apps may be in the early development stages and may not have been adequately used or tested by users. Among the 2,772 searched apps, 17 apps were included in the study.

Data Collection and Evaluation

Two researchers independently participated in the data collection process. Basic information on the app were collected, which included the app version, last updated date, number of downloads, user rating, number of raters, and price of the app. Additional information on the app were also collected, including the target group of the app, affiliations, and existence of advertisements. The secondary features provided by the dietary supplement apps were examined in the following seven features: customized dietary supplement recommendations, login function, share function, existence of app community within the app, pill taking reminder function, recording of current medications, and product comparison service. Apps were also assessed for referencing of the information provided and involvement of healthcare professional(s) in the development of the app.

App Quality Assessment Using MARS

The included apps were evaluated for quality through MARS.²⁰⁾ MARS is a multidimensional, expert-based mHealth app quality rating scale which has been utilized in various mHealth fields.^22-27) MARS evaluates mHealth apps based on 17 items in four objective dimensions; engagement (5 items), functionality (4 items), aesthetics (3 items), and information (7 items). The rating scale utilizes a 5-point Likert scale from 1 (inadequate) to 5 (excellent), and N/A is considered valid in specified items that are considered not applicable. In addition, there is a subjective quality assessment section where the evaluators subjectively evaluate the app on 4 items (app recommendation intention, expected frequency of app use, intention to pay for the app, and overall star rating of the app). Two independent evaluators majoring in pharmacy evaluated the contents of the final selection of the apps according to the evaluation tool from August 27, 2021 to September 2, 2021. Prior to the assessment, three researchers completed training on the use of MARS²⁸⁾ and conducted pilot exercises on three apps to ensure the evaluation results were consistent between the raters. Through the pilot exercises, it was determined that a more objective standard was necessary for MARS items #15 (quality of information) and #16 (quantity of information), which were regarding the information provided on dietary supplements. For consistency, the researchers selected five common dietary supplements and evaluated the quality and quantity of information provided on these supplements: vitamin D (also referred to as cholecalciferol or ergocalciferol), milk thistle (also referred to as silymarin), coenzyme Q10 (also referred to as ubidecarenone), echinacea, and ginkgo (biloba or leaf). The quality and quantity of information was determined in comparison to information provided by the Korean Ministry of Food and Drug Safety²⁹⁾ and/or Korean Integrated Medical Information System (KIMS),³⁰⁾ a Korean drug search engine. Two researchers independently participated in the MARS rating, and a third researcher was involved in cases of uncertainty, which was resolved through discussion among the researchers. Mean MARS score between the two researchers were used for final analysis.

Statistical Analysis

After calculating the mean value for each MARS item, the overall app quality score was evaluated by calculating the mean value for each of the four objective dimensions, and the overall subjective app quality was evaluated by calculating the mean value of the items in the subjective dimension, as recommended by the developers.²³⁾ In addition, correlation coefficients were calculated to examine the correlation between the MARS score and user star rating, number of raters, number of downloads, evaluators’ subjective evaluation, and number of secondary features. A p-value less than 0.05 was considered statistically significant. The Cronbach’s alpha value of this study was 0.90. All analyses were performed using the SAS version 9.4 (SAS 9.4, Cary, NC. SAS Institute Inc.).

Results of App selection

A total of 2,772 apps were identified through search terms in the app database, which included 2,372 apps on the Google Play Store and 400 apps on the Apple App Store. 690 duplicated apps and 94 apps that were not retrievable on app stores were excluded. A total of 1,988 apps were screened through the inclusion/exclusion criteria, and 1,947 apps were initially excluded. Forty-one apps were downloaded for further evaluation of the inclusion/exclusion criteria, and 24 apps were additionally excluded. Finally, 17 apps were included in the study (Fig. 1).

Fig. 1. Flow chart of the application selection process.

Characteristics of included apps

The characteristics of the apps included in the analysis are as follows. Of the 17 apps, 12 apps were compatible with both Android and iOS, whereas five apps supported Android only. All the apps clearly identified the developer and were updated at least twice. The mean user rating of all the included apps were 4.09 (standard deviation (SD): 0.56) out of 5. According to the star ratings of app users, ‘Medicinal Herbs & Supplements’ had the lowest rating of 3.2, and ‘Personalized Nutrition (Pulmuone Health & Living)’ had the highest rating with 4.9 points (Table 2).

Table 2

Characteristics of included apps

App number and name		Platform (Android/ iOS)	Developer	Involvement of healthcare professionals	Referencing of information	Tested app version	Mean user star rating*	Mean MARS score*	Mean MARS subjective score*	No. of secondary features
1	Drugs.com Medication Guide	Both	Drug.com	RPh, PharmD, MD	Always	2.12.1	4.6	3.81	4.33	6
2	Medicinal Herbs & Supplements	Android	Great_Apps	-	No reference	2.5	3.2	3.13	2.67	1
3	Vitamins, Minerals, Nutrients for immunity (Free)	Android	Medical Group Soft	-	No reference	3.6.6	4.5	3.10	3.67	1
4	Personalized Nutrition (Pulmuone Health & Living)	Both	Pulmuone Health & Living	Nutritionist, Pharmacists	Sometimes	1.3	4.9	3.12	2.50	2
5	Health Keeper	Both	Minwise Co., Ltd	-	No reference	1.1.0	3.5	2.76	2.17	2
6	Health Charging Station	Android	Hndsolution	-	Sometimes	1.0.15	3.7	2.88	2.17	5
7	Health-one-know	Both	UBcare Co., Ltd	Prof. in Clinical Nutrition	Always	1.6.2	4.2	3.26	2.83	6
8	Guntoo (Transparent Dietary Supplement)	Both	Econy	Prof. in Food engineering, Nutritionist, Pharmacists, etc.	Sometimes	3.2	3.5	3.55	3.83	5
9	Dr.milligram	Both	Dr.milligram	MD	No reference	4.5	4.1	3.20	2.83	2
10	Drug info	Android	Druginfo	Pharmacists	No reference	4.0.0	3.8	3.15	3.00	0
11	My puzzle	Both	NUTRI-1 Co., Ltd.	MD, Pharmacists, Ph.D. in Food and nutrition	Sometimes	1.0.4	4.8	3.41	3.17	4
12	My herb	Both	KHERB	-	Sometimes	1.1.4	3.5	3.30	3.17	6
13	Ayak	Both	Ayak Inc.	Pharmacists	No reference	1.0.8	4.5	3.60	3.50	4
14	Al-yak Mul-yak (Pills and Potions)	Both	Howtoyak Inc.	Nutritionist, Pharmacists	Sometimes	1.0.5	4.7	3.48	2.83	4
15	Welcoach	Android	Welcoach	-	Always	1.0.22	4.4	2.93	2.33	2
16	Pillycare	Both	Carewith	PharmD	Sometimes	1.1.3	3.3	3.76	3.83	4
17	Homespick	Both	InnomaxGlobal.co.Ltd	-	No reference	1.1.8	4.3	3.29	3.67	6
Mean (Standard Deviation)							4.09 (0.56)	3.28 (0.29)	3.09 (0.64)	3.53 (2.00)

*Scores out of 5.

MARS, Mobile App Rating Scale

Secondary features other than providing dietary supplement information were reviewed for each app. The apps had a mean of 3.53 (SD: 2.00) features with a minimum of 0 and a maximum of 6 features among the 7 features. Login function was the most commonly provided feature (14/17 apps), followed by customized recommendations (10/17 apps), share function and recording of current medications (9/17 apps each), pill taking reminder function (8/17 apps), product comparison services (6/17 apps), and the app community (4/ 17 apps) (Appendix Table 1).

A total of 10 apps (58.8%) indicated involvement of healthcare professional(s) in the app development, in which pharmacists (8 apps), nutrition specialists (5 apps), and medical doctors (3 apps) participated. For provision of information on dietary supplements and its source of references, only 3 apps (17.7%) always provided references, 7 apps (41.2%) sometimes provided references, and 7 apps (41.2%) did not provide any references to the source of information.

Overall app quality evaluation using MARS

The overall MARS app quality score of the 17 apps was 3.28 (SD: 0.29) (Appendix Table 2), and the overall app subjective quality score was 3.09 (SD: 0.64) (Appendix Table 3). Among the four objective dimensions, aesthetics had the highest score with a mean of 3.75 (SD: 0.36), followed by functionality with a mean of 3.65 (SD: 0.39), information with a mean of 3.14 (SD: 0.53), and engagement had the lowest score with mean of 2.56 (SD: 0.51) (Fig. 2). ‘Health Keeper’ and ‘Health Charging Station’ had the lowest mean MARS app quality score of 2.88, and ‘Drugs.com Medication Guide’ had the highest mean score of 3.81. The evaluators’ app subjective quality score was also lowest for ‘Health Keeper’ and ‘Health Charging Station’ with 2.17 points, and ‘Drugs.com Medication Guide’ was the highest with 4.33 points.

Table 3

Correlations between app characteristics and quality

	Variables	1	2	3	4	5	6	7	8	9	10	11	12	13
1	MARS total scores	1.0000
2	Engagement	0.6361*	1.0000
3	Functionality	0.5420†	-0.1053	1.0000
4	Aesthetics	0.6106*	0.4571	0.0314	1.0000
5	Information	0.7864**	0.2063	0.5506†	0.2040	1.0000
6	No. of downloads	0.4469	0.0961	0.4068	0.0261	0.5818†	1.0000
7	User star rating	0.1494	0.3141	-0.1827	-0.1486	0.2616	0.2432	1.0000
8	No. of raters	0.4522	0.1351	0.4007	0.0238	0.5618†	0.9970***	0.2501	1.0000
9	Subjective MARS scores	0.8348***	0.4671	0.5437†	0.2936	0.7993**	0.5310†	0.0968	0.523†	1.0000
10	Recommendation intention	0.8063***	0.5903†	0.5079†	0.3351	0.6151*	0.3325	0.1075	0.3417	0.9010***	1.0000
11	Expected app use frequency	0.7395**	0.4540	0.3856	0.2781	0.7269**	0.5604†	0.1489	0.5466†	0.9495***	0.8124***	1.0000
12	Evaluators’ star rating	0.7381**	0.2198	0.6191*	0.1841	0.8438***	0.5409†	-0.0082	0.5254†	0.8704***	0.6457*	0.7467**	1.0000
13	No. of secondary features	0.4942†	0.7586**	-0.1264	0.2892	0.2554	0.2488	0.0563	0.2837	0.3932	0.4127	0.4578	0.1763	1.0000

Mobile App Rating Scale (MARS)

***p< .0001, **p<0.001, *p<0.01, ^†p<0.05

Fig. 2. Mobile Application Rating Scale (MARS) scores by dimensions. The overall MARS app quality score of the 17 apps was 3.28 (Standard Deviation (SD): 0.29). Among the four objective dimensions, aesthetics had the highest score with a mean of 3.75 (SD: 0.36), followed by functionality with a mean of 3.65 (SD: 0.39), information with a mean of 3.14 (SD: 0.53), and engagement had the lowest score with mean of 2.56 (SD: 0.51).

Relationships Between App Characteristics and Quality

Relationships between the overall MARS score, four MARS dimensions, app characteristics, and subjective quality dimensions were determined using correlations (Table 3). Total MARS score was positively correlated to all four dimensions in the order of information, engagement, aesthetics, and functionality. Among the four dimensions, functionality and information dimensions were positively correlated. The number of app downloads and the number of app raters had a significant positive correlation with information dimension with a correlation coefficient of 0.582 (p=0.0143) and 0.562 (p=0.0189), respectively. The user star rating could not confirm a statistically significant correlation with the MARS dimensions. The subjective MARS score was positively correlated to the following variables with the correlation coefficient in the order of total MARS score, information dimension, functionality dimension, the number of app downloads, and the number of raters. Among the subjective quality assessment items, app recommendation intention and MARS total score and all MARS dimensions except for aesthetics had statistically significant positive correlation, and the correlation coefficient was in the order of total MARS score, information, engagement, and functionality. In addition, a subjective quality item, expected frequency of app use had a statistically significant positive correlation with the total MARS score, information dimension, the number of app downloads, and the number of raters. Furthermore, the subjective star rating also had a statistically significant positive correlation in the order of information dimension, total MARS score, functionality dimension, the number of app downloads, and the number of raters. The number of secondary features had a statistically significant positive correlation with the engagement dimension with a correlation coefficient of 0.759 (p=0.0004), followed by total MARS score with a correlation coefficient of 0.494 (p=0.0438).

This study systematically examined all dietary supplement apps released in English and Korean to extract general and functional characteristics and to evaluate the quality of the available apps using MARS. Furthermore, this study aimed to evaluate the dimensions related to the quality of the apps and to provide directions for improvement of dietary supplement apps. The findings have several points of discussion.

First, a gap in the ratings between app users and the MARS evaluators were observed.³¹⁾ Of the 17 included apps, the mean user star rating was 4.09 out of 5, while the mean MARS app quality score was 3.28 and the subjective quality score was 3.09. This result indicates lack of high quality dietary supplement apps on the market and need for user discretion when selecting potential dietary supplement apps based on user ratings. While significant relationship between the user star ratings and the MARS dimensions were not identified, it was interesting that the number of app downloads and number of app raters had significant positive correlation with the information dimension. While the user star ratings do not necessarily correlate to the quality of the apps,²⁰⁾ information is an important aspect for app use, and accessibility to many high quality dietary supplement apps is necessary.

Second, among the MARS dimensions, functionality and aesthetics had high scores, and the score deviations between the apps were not large. In addition, in the correlation analysis, aesthetics did not show a significant correlation with the evaluators’ subjective evaluation results. This result indicates that it requires more than a certain level of function and interface to be released as an app. It is also the result of adding evidence that functionality and aesthetics alone cannot make a high-quality app.³²⁾

Additionally, according to the evaluation score for each MARS dimension, engagement had the lowest MARS score, and the deviation between apps was high. Furthermore, information dimension showed the highest deviation with the next lowest MARS score. These results of engagement and information can degrade the motivation and persistence of the app use, which can hinder the app use for consumers.³³⁾ Engagement score significantly increased as the number of secondary features of the app increased, suggesting that engagement can be improved through supplementing secondary features in the app. Customized recommendation of dietary supplements based on user input of one’s health status was an interesting and commonly provided secondary features of the apps identified in this study, although the algorithm and its reliability of the accuracy of recommendations have not been evaluated. Other secondary features, such as the login function, app community within the app, recording of current medications, pill taking reminder function, and product comparison services were also commonly provided, which could increase user engagement and encourage continued use of the app.

The most notable result in this study was that information dimension score was rather low. Although the main aim of the dietary supplement apps are typically to deliver information to users, low information quality was identified in MARS-based scores. Low information scores in MARS have also been identified in several previous app contents analysis studies.^22,34) For dietary supplement apps, this result suggests that apps may provide insufficient or inaccurate information on dietary supplements, potentially providing unsafe information to app users. In fact, 41.2% of the apps did not provide any reference to the source of information provided within the app. According to the mean score for each item that constitutes information in the MARS dimension, the accuracy and goals of app description were above the mean, but the quality of information, amount of information, visual information, and credibility of the source of information were below the mean. In particular, credibility of information showed the lowest mean score. Even in previous studies related to dietary supplement search using the Internet,³⁵⁾ low reliability of information was a problem. As the same problem occurs even when moving to a mobile environment, it seems that involvement of healthcare professionals in the development or peer review process of the app would be important in providing reliable information on dietary supplements in a digital environment. Through correlation analysis in this study, information seemed to be related to the quality of the app the most. Previous studies also suggest that the overall quality of the app can be improved by increasing information quality. Therefore, providing evidence-based and up-to-date information is essential for a high-quality dietary supplement app.³⁴⁾

As a result of the study, while various dietary supplement apps are readily available for consumers, the result of the apps’ quality evaluation using MARS suggested that the overall quality of the apps were not high, and user ratings do not necessarily reflect the quality of the apps. Especially, quality on engagement and information dimensions were low, which suggest potential risk in user safety and insufficient motivation of continued use of the apps for users. Therefore, it is necessary to improve the information quality of the apps to achieve the original goal of the apps and ensure user safety, and to increase engagement quality of the apps by installing secondary features to motivate continued use of the apps.

This study is a systematic review and quality evaluation of dietary supplement apps conducted for the first time to the best of our knowledge. This study also tried to minimize bias by having two independent evaluators for app evaluation. However, there are some limitations to the study. First, since we limited the apps to those developed in English or Korean, many of the foreign apps may have been excluded from this study, potentially limiting generalizability. Second, the research was conducted over a limited period of time, and the characteristics of the apps presented in this study may be limited to reflecting the apps identified during the time of the study, as the app trends and characteristics continue to change and evolve over time.

The dietary supplement apps need to be developed and maintained at a higher level of quality to provide safe and reliable information to consumers. Currently available apps indicated the need to improve information and engagement dimensions to ensure safe and continued use by the users. In order to achieve this, involvement of healthcare professional(s) in the development and/or management of the apps with adequate referencing of information and adding secondary features for user engagement can be helpful.

The authors have no conflicts of interest to declare with regards to the contents of this study.