|Year : 2021 | Volume
| Issue : 3 | Page : 173-179
The reliability and validity of the Turkish version of the telehealth usability questionnaire and the telemedicine satisfaction questionnaire in patients with multiple sclerosis
Fatih Ozden1, Mehmet Özkeskin2, Zübeyir Sarı3, Özgül Ekmekçi4, Nur Yüceyar4
1 Department of Elderly Care, Kùyceğiz Vocational School of Health Services, Muğla Sıtkı Koçman University, Muğla, Turkey
2 Department of Physiotherapy and Rehabilitation, Faculty of Health Sciences, Ege University, İzmir, Turkey
3 Department of Physiotherapy and Rehabilitation, Faculty of Health Sciences, Marmara University, İstanbul, Turkey
4 Department of Neurology, Faculty of Medicine, Ege University, İzmir, Turkey
|Date of Submission||05-Mar-2021|
|Date of Decision||26-Mar-2021|
|Date of Acceptance||31-Mar-2021|
|Date of Web Publication||20-Sep-2021|
Department of Elderly Care, Kùyceğiz Vocational School of Health Services, Muğla Sıtkı Koçman University, Kùyceğiz 48800, Muğla
Source of Support: None, Conflict of Interest: None
Objective: The aim of the study was to translate and cross-culturally adapt the Telehealth Usability Questionnaire (TUQ) and the Telemedicine Satisfaction Questionnaire (TSQ) into Turkish and also to analyze the reliability and validity of both questionnaires. Materials and Methods: A total of 107 multiple sclerosis (MS) patients were recruited. The department clinician monitored all participants with telemedicine for 4 years. Internal consistency was evaluated with Cronbach's alpha coefficient. The test–retest reliability was calculated with intraclass correlation coefficient by analyzing the scores of retested 52 patients 1 week later. The construct validity was examined by Pearson's correlation coefficient (r). Besides, the internal consistency for the subscores of the TUQ and exploratory factor analysis of the TSQ was analyzed. Results: The mean age was 40.5 ± 11.0 years. Internal consistency of all items and the total score of the TUQ were excellent (>0.80; ranged: 0.976–0.979). On the other hand, the internal consistency of all items and total score of the TSQ was excellent, either (>0.80; ranged: 0.969–0.973). The reproducibility of the total score for the TUQ was excellent (>0.80). The test–retest reliability of all items and the total score of the TSQ were within limits ranging from acceptable to excellent (0.617–0.860). The reliability of the total score for the TSQ was excellent (>0.80). The internal consistency of all subscales of the TUQ was excellent (>0.80). The correlation between TUQ and TSQ was strong (r = 0.882, P < 0.001). The factor loading scores of the TSQ were high (0.814–0.919). Conclusions: The Turkish version of the TUQ and the TSQ are valid and reliable in MS patients.
Keywords: Multiple sclerosis, psychometric properties, Telehealth Usability Questionnaire, Telemedicine Satisfaction Questionnaire, Turkish version
|How to cite this article:|
Ozden F, Özkeskin M, Sarı Z, Ekmekçi &, Yüceyar N. The reliability and validity of the Turkish version of the telehealth usability questionnaire and the telemedicine satisfaction questionnaire in patients with multiple sclerosis. Neurol Sci Neurophysiol 2021;38:173-9
|How to cite this URL:|
Ozden F, Özkeskin M, Sarı Z, Ekmekçi &, Yüceyar N. The reliability and validity of the Turkish version of the telehealth usability questionnaire and the telemedicine satisfaction questionnaire in patients with multiple sclerosis. Neurol Sci Neurophysiol [serial online] 2021 [cited 2022 Jan 26];38:173-9. Available from: http://www.nsnjournal.org/text.asp?2021/38/3/173/326291
| Introduction|| |
During the past decade, the increasing use of technology has significantly enhanced remote evaluation, treatment, and rehabilitation opportunities. Remote health services, commonly termed as “telehealth” or “telemedicine,” could be conducted in several designs with various hardware and software., Telephone calls, messaging, video conferencing, virtual web-based platforms, or sensor-based applications provide essential convenience for patients who could not access face-to-face health services due to geographic, social, secular, and financial barriers. Telehealth services are particularly essential for individuals with neurological diseases.,
The changing and unpredictable nature of multiple sclerosis (MS) due to relapsing and remitting periods entails the patient follow-up process even more difficult. Besides, approximately 30% of MS patients do not have access to the treatment prescribed by a neurologist or other specialists. The management of MS requires frequent changes in the treatments and the coordinated implementation of particular care interventions by the multidisciplinary team. Telehealth services could facilitate the long-term multidisciplinary management of people with MS beyond clinical settings. Besides, remote services could provide similar opportunities to face-to-face consultation for geographically and economically disadvantaged patients. By monitoring the functional status (e.g., telephone-based Expanded Disability Status Scale (EDSS), video-conferencing-based performance tests), the drug and exercise prescription, and the patient's side effects, telemedicine applications increase the utility, satisfaction, and quality of the health-care services.,, Therefore, more emphasis has been placed on remote health services for MS patients in recent years.
The outcomes of telemedicine research to MS patients have been presented in comprehensive reviews during the past few years., Telehealth has been shown to be beneficial in enhancing treatment, exercise participation, and patient satisfaction. Currently, there is a quantity of evidence from several studies on the clinical effectiveness of telerehabilitation in improving functional activities, fatigue, and quality of life in patients with MS. Telemedicine has been demonstrated to be essential for treating motor symptoms in individuals with MS and for the continuity of treatment. In the recent randomized controlled trial, telemedicine services have been shown to be a cost-effective method of providing care for individuals with MS.
On the other hand, the level of expectation, satisfaction, and usability of telemedicine services are also essential in terms of the quality of the service provided to patients with MS.,, Subjectively presenting patients' opinions about telehealth service with standardized Patient-Reported Outcome Measures (PROMs) increases the efficiency of the remote care. Since telemedicine technologies include updatable hardware and software, the satisfaction, usefulness, feasibility, and usability should be evaluated periodically, whether the remote health services are equivalent to face-to-face interviews or better.
Indeed, System Usability Scale has been widely used for many years to evaluate the usability of several software and hardware systems. However, several evaluation tools have been developed in the past 20 years to evaluate satisfaction and usability in telehealth services, specifically. Our study aims to translate, cross-culturally adapt the Telehealth Usability Questionnaire (TUQ) and Telemedicine Satisfaction Questionnaire (TSQ) into Turkish and analyze both questionnaires' psychometric properties.
| Materials and Methods|| |
Translation and adaptation process
Permission was received from the original questionnaire developers to demonstrate the Turkish cross-cultural adaptations and psychometric properties of TUQ and TSQ. The questionnaires were translated and cross-culturally adapted in a total of five phases with internationally accepted translation procedures., In the forward-translation phase, the original English questionnaire was translated into Turkish independently by four academicians who are native Turkish speakers and experts in English. In the second stage, members of the same committee discussed the translations and the relevant correction notes, then synthesized the translations. At this stage, the term “telerehabilitation service” was preferred instead of “system” in TUQ. On the other hand, in TSQ, the term “clinician” was used alternatively to “health-care provider.” A single prefinal version of the questionnaires was accomplished by synthesizing the translations. In the third stage, the prefinal version questionnaires were back-translated into Turkish independently by two native English-speaking translators. The original English questionnaires were compared with the back-translated versions in the fourth phase, whether the Turkish prefinal versions were conceptually and linguistically accurate. In the last phase, the pilot test was used to question whether the questionnaire was suitable in terms of comprehensibility. A pretest was carried out with thirty Turkish-speaking individuals with a 5-point Likert-type scale. Comprehensibility was excellent in the pilot study. No additional change was required. Finally, the final version of TUQ and TSQ has been established.
Sample size estimation
The sample size was calculated based on both power analysis and statistical formula. First, Fayers and Machin suggested that the sample size should be at least 100 and also a minimum of five times of the questionnaires in cross-cultural adaptation studies. TUQ and TSQ have 21 and 14 items, respectively. Considering the numerous items of the TUQ questionnaire, a total of 105 participants were required for the psychometric analysis. The minimum sample for the reproducibility was calculated by G*power-3 software (Heinrich-Heine-Universität, Düsseldorf, Germany) with an effect size of 0.45, a probability of error (α) = 0.05, and the power (1−ß) of 0.80. Forty-six patients were required for reproducibility. Consequently, 107 MS patients were assessed in the first evaluation, then 52 patients were refilled the TUQ and TSQ 1 week later for the retest reliability.
A prospective cross-sectional study was carried out in individuals with MS, followed by “Ege” University, Department of Neurology. The permission for the translation for the Turkish version of the TUQ and the TSQ was acquired from the developers of the original questionnaires. Informed consents of the patients were obtained. The study protocol was approved by the ethics committee of “Ege” University (No: “21-1.1T/8”).
A neurologist considered the MS diagnosis of the patients according to the revised Mc Donald criteria. The department clinician monitored all participants with telemedicine for 4 years. Within the scope of telemedicine, patients were rehabilitated remotely (telephone, messaging, e-mail, video-based consultation) in terms of medication, exercise prescription, treatment consultancy, side effects, and symptom severity. The inclusion criteria of the study were (1) Turkish-speaking patients, (2) patients aged ≥18 years, and (3) no relapse for a minimum of 1 month. The exclusion criteria of the study were (1) patients in the relapse period, (2) bedridden individuals, (3) comorbid conditions affecting functionality, (4) cognitive disorders, and (5) patients who do not give consent.
The sociodemographic and physical characteristics of the patients were questioned. One hundred and seven patients were evaluated in the first session. Participants were filled the TUQ and TSQ. Then, 1 week later, 52 patients were re-filled both questionnaires in the second assessment session.
Telehealth Usability Questionnaire
TUQ consists of 21 items, evaluating treatment given through the remote rehabilitation service, software, or system. The survey was developed by Parmanto et al. in 2016. The survey addresses six factors: usability, ease of use and learnability, interface quality, interaction quality, reliability, and satisfaction. TUQ uses 7-point Likert type scale (1 = disagree and 7 = agree). The total score is calculated by summing the 21 items.
Telemedicine Satisfaction Questionnaire
TSQ consists of 14 items. The satisfaction level of the patients with the telerehabilitation service is evaluated. The questionnaire was developed by Yip et al. in 2003 and is scored with a 5-point Likert scale (1 = strongly disagree and 5 = strongly agree). The total score ranges from 14 to 70.
All data collected from the research were statistically analyzed using SPSS for Windows v25.0 (SPSS Inc., Chicago, IL, USA). Mean and the standard deviation were given for the quantitative data. Percentages were presented for the qualitative data. The Shapiro–Wilk test was used to test whether the data of the sample showed a homogeneous distribution. In addition, 95% confidence intervals (CI) were accepted for the correlation coefficients.
The Cronbach alpha coefficient was estimated to investigate the internal consistency of the TUQ and TSQ. A greater alpha coefficient means that TUQ and TSQ have a better consistency. An alpha of ≥0.6 was considered acceptable; ≥0.8 was rated excellent consistency. Shrout and Fleiss's conventions of the two-way random-effect model single-measure reliability analysis were preferred. The intraclass correlation coefficient (ICC: 95% CI) was used to describe reproducibility. ICC was estimated for the total score and items of the TUQ and TSQ. An ICC ≥0.8 signifies excellent test–retest reliability. The minimal detectable change (MDC95) was calculated to explain the clinically significant difference rather than measurement error or bias. The standard error of measurement (SEM95) and MDC95 were calculated with the method provided below.
Formula (1) MDC95 = 1.96 ∗ SEM∗√2.
Formula (2) SEM95 = SD∗√(1 − ICC).
The construct validity of TUQ and TSQ was examined by Pearson's correlation coefficient (r). The total of the TUQ and all items were compared with TSQ. A high correlation coefficient was presumed in terms of convergent validity. The correlation was deemed strong if the coefficient was >0.5; medium if the coefficient was between 0.5 and 0.35; and low if the value was smaller than 0.35. Besides, the internal consistency for the subscores of the TUQ was calculated. In addition, exploratory factor analysis (principal component and varimax rotation) was carried out to investigate the factor structure of the TSQ. The Kaiser–Meyer–Olkin (KMO) test was applied to estimate sample competence, and the Bartlett test of sphericity was used to explore the correlation matrix.
| Results|| |
A total of 107 patients (67 women and 40 men) with a mean age of the 40.5 ± 11.0 years were included in the study. MS duration of the patients was 9.72 ± 5.98 years. The absolute values of the patients are given in [Table 1]. In the pilot study conducted with 30 Turkish-speaking individuals for the comprehensibility of TUQ and TSQ, both questionnaires were excellently understandable in linguistic and cultural concepts. For this reason, no further changes were required after the pretest.
The Cronbach's alpha coefficient was calculated for the items and total scores of the TUQ and TSQ. Internal consistency of all items and the total score of the TUQ were excellent (>0.80; ranged 0.976–0.979). On the other hand, the internal consistency of all items and total score of the TSQ was excellent, either (>0.80; ranged 0.969–0.973). The ICC of the TUQ items and the total score were ranged from 0.628 to 0.887. The reproducibility of the total score for the TUQ was excellent (>0.80) [Table 2]. The test–retest reliability of all items and the total score of the TSQ were within limits ranging from acceptable to excellent (0.617–0.860) [Table 3]. The reliability of the total score for the TSQ was excellent (>0.80). The SEM95 and the MDC95 of the TUQ were 8.14 and 22.56, respectively. In addition, the SEM95 and the MDC95 of the TSQ were 5.19 and 14.38, respectively.
|Table 2: Test-retest reliability and internal consistency for the Telehealth Usability Questionnaire|
Click here to view
|Table 3: Test-retest reliability and internal consistency for the Telemedicine Satisfaction Questionnaire|
Click here to view
The validity of the TUQ was demonstrated with the internal consistency of subscores and a correlation with TSQ. The internal consistency of all subscales of the TUQ was excellent (>0.80) [Table 4]. The correlation between TUQ and TSQ was strong (r = 0.882, P < 0.001. In addition, there was a strong relationship between the subscales of the TUQ and the total score of the TSQ (r > 0.5, P < 0.001) [Table 5]. Construct validity of the TSQ was also calculated using factor analysis. Sample adequacy was evaluated for the analysis. The KMO test was calculated (0.946), and the significance level of Bartlett's Test of Sphericity was < 0.001. [Table 6] presents the TSQ items their factor loadings on principal component analysis-derived measures. The extraction communality scores were high (0.814–0.919). One factor was extracted with eigenvalues of 1 accounting for 79.37% of the variance. All items of the TSQ were loaded on factor 1. The factor analysis results proved that the items of the TSQ composed a uniform group.
|Table 4: Internal consistency of five factors of the Telehealth Usability Questionnaire (n=107)|
Click here to view
|Table 5: Correlation between Telehealth Usability Questionnaire and Telemedicine Satisfaction Questionnaire (n=107)|
Click here to view
|Table 6: Factor loadings of the Telemedicine Satisfaction Questionnaire (n=107)|
Click here to view
| Discussion|| |
The present study was purposed to demonstrate the translation, cross-cultural adaptation, reliability, and validity of the TUQ and TSQ in patients with MS who were followed up 4 years with telemedicine by the neurology department. Consequently, the Turkish version of the TUQ and TSQ was proved to be valid and reliable PROM for evaluating Turkish-speaking patients with MS. Given the lack of Turkish assessment tools to evaluate telehealth services, both questionnaires are unique assessment tools for Turkish-speaking patients. The translation and adaptation of the TUQ and TSQ into Turkish are also essential for nearly 3 million Turkish residents living in other European Union countries.
Telemedicine services should be updated periodically by inspecting their usability and quality with PROMs. TUQ has been developed and widely used for years. In addition to the original questionnaire in English, the Spanish version was also culturally adapted recently. On the other hand, TSQ is available only in its original English version. Considering the rapid increase in telemedicine and telerehabilitation in recent years and the importance of remote health services in the COVID-19 process, the Turkish version of both questionnaires could significantly contribute to clinical practice and other case–control or intervention studies.
In the translation adaptation phases of the TUQ and TSQ, the term “telerehabilitation service” was revised as “system” in TUQ. The translation committee predicted this technical change with the idea that telemedicine service might not be provided only through the “system.” Besides, in TSQ, the term “clinician” was preferred instead of “health-care provider.” The translation committee carried out this change, because the “clinician” identifies the potential telemedicine provider working in all health sciences. The committee aimed to increase compliance with the interprofessional use of the questionnaire.
The pilot study of the TUQ and TSQ revealed that both tools were proper in terms of comprehensibility. Accordingly, no other modifications were required of the pretest. In the Spanish version study of TUQ, a pretest was stated to be conducted, but technical details were not presented.
The internal consistency for the total score and all items of the TUQ and TSQ was excellent (α > 0.80). The results demonstrated that TUQ and TSQ are consistently evaluating the usability and satisfaction of the telemedicine services, respectively. In addition, the items of both questionnaires are compatible within themselves. Besides, the Cronbach's alpha of all subscores of the TUQ (usefulness, ease of use, effectiveness, reliability, and satisfaction) was highly consistent (α > 0.80). Five factors of the questionnaire clearly represent their assessment purpose. The Spanish version of the TUQ only demonstrated the content validity that did not provide any internal consistency analysis. TUQ development study was presented the Cronbach's alpha scores of the subscales. The standardized alpha values of the usefulness, ease of use, effectiveness, reliability, and satisfaction scores were ranged from 0.81 to 0.93. Their calculation results were conformed to our study (α > 0.80). The TSQ development study was only presented the total scores' alpha value (0.93). The results of Yip et al. were also similar to our study. All studies' results have demonstrated a consensus about the internal consistency of the TUQ and TSQ.
The test–retest reliability for the total scores of the TUQ and TSQ was excellent (ICC >0.80). In addition, all items of both questionnaires have both moderate to excellent reproducibility (>0.60). Only, the development study of the TSQ calculated ICC (0.43). The authors of the study explained the low coefficient by stating telemedicine as a relatively new era. Considering that Yip et al. conducted their development and psychometric property analysis study in 2003, the low coefficient might be acceptable in terms of pragmatical concept. In our study, the ICC did not drop below 0.60. This score has greatly increased the questionnaires' reliability as an advantage of the widespread use of technology and telemedicine service for our study.
One of our study's unique features is that SEM95 and MDC95 values are presented for the first time for TUQ and TSQ. The SEM95 and the MDC95 of the TUQ were 8.14 and 22.56, respectively. In addition, the SEM95 and the MDC95 of the TSQ were 5.19 and 14.38, respectively. Through MDC95, the usability and satisfaction levels of the telemedicine service of the patients could be monitored quantitatively. MDC95 represents a remarkable minimal level of an individual's view of telemedicine without bias or measurement error.
Construct validity was analyzed by comparing the TUQ and TSQ scores. The correlation between TUQ and TSQ was strong (r = 0.882, P < 0.001). In addition, there was a strong relationship between the subscales of the TUQ and the total score of the TSQ (r > 0.5, P < 0.001). No other studies preferred correlation coefficient in order to demonstrate validity. In the early 2000s, when development studies were conducted, the only questionnaire that could be compared was the System Usability Scale. However, the authors did not choose to carry out this comparison. The present study was preferred to adapt two common telemedicine-specific questionnaires and compare them with each other instead of the System Usability Scale. According to the calculation results of our study, both questionnaires were found to be valid.
Construct validity of the TSQ was also calculated using factor analysis. The extraction communality scores were high (0.814–0.919). All items of the TSQ were loaded on factor 1. The factor analysis results proved that the items of the TSQ composed a homogenous group for all items. The development study of the TSQ showed a 3-factor structure. However, since the scale does not have subscores, our result showing a single factor structure could be accepted because TSQ has only a total score. For instance, Yip et al. stated that one of the factors of the TSQ was “similarity to face-to-face encounter.”
Some limitations of the study should also be acknowledged. Responsiveness analysis was not analyzed in the present research. Besides, the “System Usability Scale” was not used to demonstrate validity. Thorough with a 10-item System Usability Questionnaire, the construct validity could be further explained. However, asking the participants additional questions may be burdensome. It can also disrupt precise data collection.
| Conclusions|| |
The Turkish version of the TUQ and TSQ are reliable and valid tools to assess usability and satisfaction in patients with MS. Both questionnaires are reliable and valid PROMs for MS patients.
Financial support and sponsorship
Conflicts of interest
There are no conflicts of interest.
| References|| |
Galea MD. Telemedicine in rehabilitation. Phys Med Rehabil Clin N Am 2019;30:473-83.
Kruse CS, Krowski N, Rodriguez B, Tran L, Vela J, Brooks M. Telehealth and patient satisfaction: A systematic review and narrative analysis. BMJ Open 2017;7:e016242.
Hatcher-Martin JM, Adams JL, Anderson ER, Bove R, Burrus TM, Chehrenama M, et al
. Telemedicine in neurology: Telemedicine Work Group of the American Academy of Neurology update. Neurology 2020;94:30-8.
Chumbler NR, Quigley P, Li X, Morey M, Rose D, Sanford J, et al
. Effects of telerehabilitation on physical function and disability for stroke patients: A randomized, controlled trial. Stroke 2012;43:2168-74.
Amatya B, Galea MP, Kesselring J, Khan F. Effectiveness of telerehabilitation interventions in persons with multiple sclerosis: A systematic review. Mult Scler Relat Disord 2015;4:358-69.
Cavalera C, Pagnini F, Rovaris M, Mendozzi L, Pugnetti L, Garegnani M, et al
. A telemedicine meditation intervention for people with multiple sclerosis and their caregivers: Study protocol for a randomized controlled trial. Trials 2016;17:4.
Robb JF, Hyland MH, Goodman AD. Comparison of telemedicine versus in-person visits for persons with multiple sclerosis: A randomized crossover study of feasibility, cost, and satisfaction. Mult Scler Relat Disord 2019;36:101258.
Minden SL, Hoaglin DC, Hadden L, Frankel D, Robbins T, Perloff J. Access to and utilization of neurologists by people with multiple sclerosis. Neurology 2008;70:1141-9.
Rogante M, Grigioni M, Cordella D, Giacomozzi C. Ten years of telerehabilitation: A literature overview of technologies and clinical applications. NeuroRehabilitation 2010;27:287-304.
Hailey D, Roine R, Ohinmaa A, Dennett L. Evidence of benefit from telerehabilitation in routine care: A systematic review. J Telemed Telecare 2011;17:281-7.
Syed MJ, Shah Z, Awan S, Wasay M, Fredrikson S. Telephone validation of an Urdu translated version of the extended disability severity scale in multiple sclerosis patients. Mult Scler Relat Disord 2021;48:102684.
Domingues RB, Mantese CE, Aquino ED, Fantini FG, Prado GF, Nitrini R. Telemedicine in neurology: Current evidence. Arq Neuropsiquiatr 2020;78:818-26.
Yeroushalmi S, Maloni H, Costello K, Wallin MT. Telemedicine and multiple sclerosis: A comprehensive literature review. J Telemed Telecare 2020;26:400-13.
Khan F, Amatya B, Kesselring J, Galea M. Telerehabilitation for persons with multiple sclerosis. Cochrane Database Syst Rev 2015;2015:CD010508.
Di Tella S, Pagliari C, Blasi V, Mendozzi L, Rovaris M, Baglio F. Integrated telerehabilitation approach in multiple sclerosis: A systematic review and meta-analysis. J Telemed Telecare 2020;26:385-99.
Parmanto B, Lewis AN Jr., Graham KM, Bertolet MH. Development of the Telehealth Usability Questionnaire (TUQ). Int J Telerehabil 2016;8:3-10.
Yip MP, Chang AM, Chan J, MacKenzie AE. Development of the telemedicine satisfaction questionnaire to evaluate patient satisfaction with telemedicine: A preliminary study. J Telemed Telecare 2003;9:46-50.
Bakken S, Grullon-Figueroa L, Izquierdo R, Lee NJ, Morin P, Palmas W, et al
. Development, validation, and use of English and Spanish versions of the telemedicine satisfaction and usefulness questionnaire. J Am Med Inform Assoc 2006;13:660-7.
Klaassen B, van Beijnum BJ, Hermens HJ. Usability in telemedicine systems – A literature survey. Int J Med Inform 2016;93:57-69.
Brooke J. Sus: A “Quick and Dirty'usability. Usability Evaluation in Industry. Taylor & Francis: London; 1996. p. 189.
Beaton DE, Bombardier C, Guillemin F, Ferraz MB. Guidelines for the process of cross-cultural adaptation of self-report measures. Spine (Phila Pa 1976) 2000;25:3186-91.
Guillemin F, Bombardier C, Beaton D. Cross-cultural adaptation of health-related quality of life measures: Literature review and proposed guidelines. J Clin Epidemiol 1993;46:1417-32.
Faul F, Erdfelder E, Lang AG, Buchner A. G*Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behav Res Methods 2007;39:175-91.
Schermelleh-Engel K, Moosbrugger H, Müller H. Evaluating the fit of structural equation models: Tests of significance and descriptive goodness-of-fit measures. Psychol Methods 2003;8:23-74.
Baumgartner TA, Chung H. Confidence limits for intraclass reliability coefficients. Meas Phys Educ Exerc Sci 2001;5:179-88.
Portney LG, Watkins MP. Foundations of Clinical Research: Applications to Practice. Upper Saddle River, NJ: Pearson/Prentice Hall; 2009.
Juniper EF. How to develop and validate a new health-related quality of life instrument. In: Quality of Life and Pharamacoeconomics in Clinical Trials. Vol. 1. Lippincott-Raven Publishers:Philadelphia; 1996. p. 49-56.
Tuğay U, Tuğay N, Gelecek N, Özkan M. Oxford shoulder score: Cross-cultural adaptation and validation of the Turkish version. Arch Orthop Trauma Surg 2011;131:687-94.
Barit A. Updated guidelines on telemedicine – A step in the right direction. S Afr Med J 2020;110:566.
Torre AC, Bibiloni N, Sommer J, Plazzotta F, Angles MV, Terrasa SA, et al
. Spanish translation and transcultural adaptation of a questionnaire on telemedicine usability. Medicina (B Aires) 2020;80:134-7.
Portnoy J, Waller M, Elliott T. Telemedicine in the era of COVID-19. J Allergy Clin Immunol Pract 2020;8:1489-91.
[Table 1], [Table 2], [Table 3], [Table 4], [Table 5], [Table 6]