Changes

Resources:Measuring Statistical Evidence Using Relative Belief (view source)

Revision as of 23:25, 24 March 2021

30 bytes added , 4 years ago

no edit summary

Line 48: Line 48:

Note that in Example 1 there are no infinities and everything is defined simply in terms of counting.

So the position taken here is that in statistical problems there are essentially no infinities and there are no continuous distributions. Infinity and continuity are employed as simplifying approximations to a

−

finite reality. This has a number of consequences, for example, any counterexample or paradox that depends intrinsically on infinity is not valid. Also, densities must be defined as limits as in f θ (x) = lim ε → 0 P θ (N ε (x))/Vol(N ε (x)) where N ε (x) is a set that shrinks nicely to x, as described in Rudin<ref>{{cite ~~res~~

+

finite reality. This has a number of consequences, for example, any counterexample or paradox that depends intrinsically on infinity is not valid. Also, densities must be defined as limits as in f θ (x) = lim ε → 0 P θ (N ε (x))/Vol(N ε (x)) where N ε (x) is a set that shrinks nicely to x, as described in Rudin<ref>{{cite book

| autore = Rudin W

| titolo = Real and Complex Analysis

Line 73: Line 73:

This supposes that the data, which hereafter is denoted by x, has been collected appropriately and so can be considered as being objective.

−

Model checking, where it is asked if the observed data is surprising for each fθ in the model, is a familiar process and so the model satisfies this principle. It is less well-known that it is possible to provide a consistent check on the prior by assessing whether or not the true value of θ is a surprising value for π. Such a check is carried out by computing a tail probability based on the prior predictive distribution of a minimal sufficient statistic (see Evans and Moshonov<ref>{{cite ~~res~~

+

Model checking, where it is asked if the observed data is surprising for each fθ in the model, is a familiar process and so the model satisfies this principle. It is less well-known that it is possible to provide a consistent check on the prior by assessing whether or not the true value of θ is a surprising value for π. Such a check is carried out by computing a tail probability based on the prior predictive distribution of a minimal sufficient statistic (see Evans and Moshonov<ref>{{cite book

| autore = Evans M

| autore2 = Moshonov H

Line 88: Line 88:

| OCLC =

| related = https://scholar.google.com/scholar?q=related:IioLHo5LRgAJ:scholar.google.com/&scioq=&hl=it&as_sdt=0,5

−

}}</ref>,<ref>{{cite ~~res~~

+

}}</ref>,<ref>{{cite book

| autore = Evans M

| autore2 = Moshonov H

Line 103: Line 103:

| OCLC =

| related = https://scholar.google.com/scholar?q=related:PuaVvOVtnmgJ:scholar.google.com/&scioq=&hl=it&as_sdt=0,5

−

}}</ref>). In Evans and Jang<ref>{{cite ~~res~~

+

}}</ref>). In Evans and Jang<ref>{{cite book

| autore = Evans M

| autore2 = Jang GH

Line 118: Line 118:

| OCLC =

| related = https://scholar.google.com/scholar?q=related:fJrrMnixVHUJ:scholar.google.com/&scioq=&hl=it&as_sdt=0,5

−

}}</ref> it is proved that this tail probability is consistent in the sense that, as the amount of data grows, it converges to a probability that measures how far into the tails of the prior the true value of θ lies. Here “lying in the tails” is interpreted as indicating that a prior-data conflict exists since the data is not coming from a distribution where the prior assigns most of the belief. In Evans and Jang<ref>{{cite ~~res~~

+

}}</ref> it is proved that this tail probability is consistent in the sense that, as the amount of data grows, it converges to a probability that measures how far into the tails of the prior the true value of θ lies. Here “lying in the tails” is interpreted as indicating that a prior-data conflict exists since the data is not coming from a distribution where the prior assigns most of the belief. In Evans and Jang<ref>{{cite book

| autore = Evans M

| autore2 = Jang GH

Line 133: Line 133:

| OCLC =

| related = https://scholar.google.com/scholar?q=related:6Cs4kQ_rD7kJ:scholar.google.com/&scioq=&hl=it&as_sdt=0,5

−

}}</ref> it is shown how this approach to assessing prior-data conflict can be used to characterize weakly informative priors and also how to modify a prior, when such a conflict is obtained, in a way that is not data dependent, to avoid such a conflict. Further details and discussion on all of this can be found in Evans<ref name=self>{{cite ~~res~~

+

}}</ref> it is shown how this approach to assessing prior-data conflict can be used to characterize weakly informative priors and also how to modify a prior, when such a conflict is obtained, in a way that is not data dependent, to avoid such a conflict. Further details and discussion on all of this can be found in Evans<ref name=self>{{cite book

| autore = Evans M

| titolo = Measuring Statistical Evidence Using Relative Belief

Line 161: Line 161:

There are of course many discussions in the statistical literature concerning the measurement of evidence. Chapter 3 of Evans<ref name=self /> contains extensive analyses of many of these and documents why they cannot be considered as fully satisfactory treatments of statistical evidence. For example, sections of that text are devoted to discussions of pure likelihood theory, frequentist theory and p-values, Bayesian theories and Bayes factors, and fiducial inference. Some of the salient points are presented in the following paragraphs together with further references.

−

Edwards<ref>{{cite ~~res~~

+

Edwards<ref>{{cite book

| autore = Edwards AWF

| titolo = Likelihood, Expanded Edition

Line 175: Line 175:

| OCLC =

| related = https://scholar.google.com/scholar?q=related:hf_5nrcD7E0J:scholar.google.com/&scioq=Edwards+A.W.F.+Likelihood,+Expanded+Edition+1992+The+Johns+Hopkins+University+Press&hl=it&as_sdt=0,5

−

}}</ref> and Royall<ref name=r26>{{cite ~~res~~

+

}}</ref> and Royall<ref name=r26>{{cite book

| autore = Royall R

| titolo = Statistical Evidence: A Likelihood Paradigm

Line 191: Line 191:

}}</ref> develop an approach to inference based upon recognizing the centrality of the concept of statistical evidence and measuring this using likelihood ratios for the full model parameter θ. A likelihood ratio, however, is a measure of relative evidence between two values of θ and is not a measure of the evidence that a particular value θ is true. The relative belief ratio for θ, defined in Section 2, is a measure of the evidence that θ is true and furthermore a calibration of this measure of evidence is provided. While these are significant differences in the two approaches, there are also similarities between the pure likelihood approach and relative belief approach to evidence. For example, it is easily seen that the relative belief ratio for θ gives the same ratios between two values as the likelihood function. Another key difference arises, however, when considering measuring evidence for an arbitrary ψ = Ψ(θ). Pure likelihood theory does not deal with such marginal parameters in a satisfactory way and the standard recommendation is to use a profile likelihood. A profile likelihood is generally not a likelihood and so the basic motivating idea is lost. By contrast the relative belief ratio for such a ψ is defined in a consistent way as a measure of change in belief.

−

In frequency theory p-values are commonly used as measures of evidence. A basic issue that arises with the p-value is that a large value of such a quantity cannot be viewed as evidence that a hypothesis is true. This is because in many examples, a p-value is uniformly distributed when the hypothesis is true. It seems clear that any valid measure of evidence must be able to provide evidence for something being true as well as evidence against and this is the case for the relative belief ratio. Another key problem for p-values arises with so-called “data snooping” as discussed in Cornfield<ref>{{cite ~~res~~

+

In frequency theory p-values are commonly used as measures of evidence. A basic issue that arises with the p-value is that a large value of such a quantity cannot be viewed as evidence that a hypothesis is true. This is because in many examples, a p-value is uniformly distributed when the hypothesis is true. It seems clear that any valid measure of evidence must be able to provide evidence for something being true as well as evidence against and this is the case for the relative belief ratio. Another key problem for p-values arises with so-called “data snooping” as discussed in Cornfield<ref>{{cite book

| autore = Cornfield J

| titolo = Sequential trials, sequential analysis and the likelihood principle

Line 205: Line 205:

| OCLC =

| related = https://scholar.google.com/scholar?q=related:pJz799PuPD8J:scholar.google.com/&scioq=&hl=it&as_sdt=0,5

−

}}</ref> where an investigator who wants to use the standard 5% value for significance can be prevented from ever attaining significance if they obtain a slightly larger value for a given sample size and then want to sample further to settle the issue. Royall<ref name=r26 /> contains a discussion of many of the problems associated with p-values as measures of evidence. A much bigger issue for a frequency theory of evidence is concerned with the concept of ancillary statistics and the conditionality principle. The lack of a unique maximal ancillary leads to ambiguities in the characterization of evidence as exemplified by the discussion in Birnbaum<ref name=r2>{{cite ~~res~~

+

}}</ref> where an investigator who wants to use the standard 5% value for significance can be prevented from ever attaining significance if they obtain a slightly larger value for a given sample size and then want to sample further to settle the issue. Royall<ref name=r26 /> contains a discussion of many of the problems associated with p-values as measures of evidence. A much bigger issue for a frequency theory of evidence is concerned with the concept of ancillary statistics and the conditionality principle. The lack of a unique maximal ancillary leads to ambiguities in the characterization of evidence as exemplified by the discussion in Birnbaum<ref name=r2>{{cite book

| autore = Birnbaum A

| titolo = On the foundations of statistical inference (with discussion)

Line 219: Line 219:

| OCLC =

| related = https://scholar.google.com/scholar?q=related:jduxSZgDaxUJ:scholar.google.com/&scioq=&hl=it&as_sdt=0,5

−

}}</ref>, Evans, Fraser and Monette<ref>{{cite ~~res~~

+

}}</ref>, Evans, Fraser and Monette<ref>{{cite book

| autore = Evans M

| autore2 = Fraser DAS

Line 235: Line 235:

| OCLC =

| related = https://scholar.google.com/scholar?q=related:7AJ153LKZZAJ:scholar.google.com/&scioq=&hl=it&as_sdt=0,5

−

}}</ref> and Evans<ref>{{cite ~~res~~

+

}}</ref> and Evans<ref>{{cite book

| autore = Evans M

| titolo = What does the proof of Birnbaum's theorem prove?

Line 249: Line 249:

| OCLC =

| related = https://scholar.google.com/scholar?q=related:WzU8WIUGg-0J:scholar.google.com/&scioq=&hl=it&as_sdt=0,5

−

}}</ref>. A satisfactory frequentist theory of evidence requires a full resolution of this issue. The book Taper and Lele<ref>{{cite ~~res~~

+

}}</ref>. A satisfactory frequentist theory of evidence requires a full resolution of this issue. The book Taper and Lele<ref>{{cite book

| autore = Taper M

| autore2 = Lele SR

Line 266: Line 266:

}}</ref> contains a number of papers discussing the concept of evidence in the frequentist and pure likelihood contexts.

−

In a Bayesian formulation the Bayes factor is commonly used as a measure of evidence. The relationship between the Bayes factor and the relative belief ratio is discussed in Section 2. It is also the case, however, that posterior probabilities are used as measures of evidence. Relative belief theory, however, draws a sharp distinction between measuring beliefs, which is the role of probability, and measuring evidence, which is measured by change in beliefs from a priori to a posteriori. As discussed in the following sections, being careful about this distinction is seen to resolve a number of anomalies for inference. Closely related to Bayesian inference is entropic inference as discussed, for example, in Caticha<ref>{{cite ~~res~~

+

In a Bayesian formulation the Bayes factor is commonly used as a measure of evidence. The relationship between the Bayes factor and the relative belief ratio is discussed in Section 2. It is also the case, however, that posterior probabilities are used as measures of evidence. Relative belief theory, however, draws a sharp distinction between measuring beliefs, which is the role of probability, and measuring evidence, which is measured by change in beliefs from a priori to a posteriori. As discussed in the following sections, being careful about this distinction is seen to resolve a number of anomalies for inference. Closely related to Bayesian inference is entropic inference as discussed, for example, in Caticha<ref>{{cite book

| autore = Caticha A

| titolo = Entropic Inference and the Foundations of Physics

Line 280: Line 280:

| OCLC =

| related = https://scholar.google.com/scholar?q=related:WJvH0aCChRsJ:scholar.google.com/&scioq=Entropic+Inference+and+the+Foundations+of+Physics&hl=it&as_sdt=0,5

−

}}</ref>,<ref>{{cite ~~res~~

+

}}</ref>,<ref>{{cite book

| autore = Caticha A

| titolo = Towards an informational pragmatic realism

Line 294: Line 294:

| OCLC =

| related = https://scholar.google.com/scholar?q=related:4ZlK3QSAe_0J:scholar.google.com/&scioq=&hl=it&as_sdt=0,5

−

}}</ref>. In entropic inference relative entropy plays a key role in determining how beliefs are to be updated after obtaining information. This is not directly related to relative belief as discussed here, although updating beliefs via conditional probability is central to the approach and so there are some points in common. Another approach to measuring statistical evidence, based on a thermodynamical analogy, can be found in Vieland<ref>{{cite ~~res~~

+

}}</ref>. In entropic inference relative entropy plays a key role in determining how beliefs are to be updated after obtaining information. This is not directly related to relative belief as discussed here, although updating beliefs via conditional probability is central to the approach and so there are some points in common. Another approach to measuring statistical evidence, based on a thermodynamical analogy, can be found in Vieland<ref>{{cite book

| autore = Vieland VJ

| titolo = Evidence, temperature, and the laws of thermodynamics

Line 310: Line 310:

}}</ref>.

−

The Dempster–Shafer theory of belief functions, as presented in Shafer<ref>{{cite ~~res~~

+

The Dempster–Shafer theory of belief functions, as presented in Shafer<ref>{{cite book

| autore = Shafer G

| titolo = A Mathematical Theory of Evidence

Line 324: Line 324:

| OCLC =

| related = https://scholar.google.com/scholar?q=related:xWNnb662qbYJ:scholar.google.com/&scioq=Shafer+G.+A+Mathematical+Theory+of+Evidence+1976+Princeton+University+Press+&hl=it&as_sdt=0,5

−

}}</ref>, is another approach to the development of a theory of evidence. This arises by extending the usual formulation of probability, as the measure of belief in the truth of a proposition, to what could be considered as upper and lower bounds on this belief. While this clearly distinguishes the theory of belief functions from relative belief, a more fundamental distinction arises from measuring evidence via a change in belief in the relative belief approach as opposed to using probability itself or bounds based on probabilities. Cuzzolin<ref>{{cite ~~res~~

+

}}</ref>, is another approach to the development of a theory of evidence. This arises by extending the usual formulation of probability, as the measure of belief in the truth of a proposition, to what could be considered as upper and lower bounds on this belief. While this clearly distinguishes the theory of belief functions from relative belief, a more fundamental distinction arises from measuring evidence via a change in belief in the relative belief approach as opposed to using probability itself or bounds based on probabilities. Cuzzolin<ref>{{cite book

| autore = Cuzzolin F

| titolo = On the relative belief transform

Line 350: Line 350:

{{q|<nowiki>Principle of evidence: If P(A | C) > P(A), then there is evidence in favor of A being true because the belief in A has increased. If P(A | C) < P(A), then there is evidence A is false because the belief in A has decreased. If P(A | C) = P(A), then there isn't evidence either in favor of A or against A as belief in A has not changed.</nowiki>}}

−

This principle suggests that any valid measure of the quantity of evidence is a function of (P(A), P(A | C)). A number of such measures have been discussed in the literature and Crupi et al.<ref>{{cite ~~res~~

+

This principle suggests that any valid measure of the quantity of evidence is a function of (P(A), P(A | C)). A number of such measures have been discussed in the literature and Crupi et al.<ref>{{cite book

| autore = Crupi V

| autore2 = Tentori K

Line 372: Line 372:

So, for example, RB(A | C) > 1 implies that observing C is evidence in favor of A and the bigger RB(A | C) is, the more evidence in favor.

−

The Bayes factor is also used as a measure of evidence. The Bayes factor BF(A | C) in favor of A being true is the ratio of the posterior to prior odds in favor of A. It is easily shown that BF(A | C) = RB(A | C)/B(Ac | C), namely, from the point of view of the relative belief ratio, the Bayes factor is a comparison between the evidence in favor of A and the evidence in favor of its negation. The relative belief ratio satisfies RB(A | C) = BF(A | C)/(1 − P(A) + P(A)BF(A | C)) and so cannot be expressed in terms of the Bayes factor itself. From this it is concluded that the relative belief ratio is a somewhat more elemental measure of evidence. As discussed in Baskurt and Evans <ref name=r1>{{cite ~~res~~

+

The Bayes factor is also used as a measure of evidence. The Bayes factor BF(A | C) in favor of A being true is the ratio of the posterior to prior odds in favor of A. It is easily shown that BF(A | C) = RB(A | C)/B(Ac | C), namely, from the point of view of the relative belief ratio, the Bayes factor is a comparison between the evidence in favor of A and the evidence in favor of its negation. The relative belief ratio satisfies RB(A | C) = BF(A | C)/(1 − P(A) + P(A)BF(A | C)) and so cannot be expressed in terms of the Bayes factor itself. From this it is concluded that the relative belief ratio is a somewhat more elemental measure of evidence. As discussed in Baskurt and Evans <ref name=r1>{{cite book

| autore = Baskurt Z

| autore2 = Evans M

Line 406: Line 406:

<center><nowiki>ψ(x) = argsupRBΨ(ψ|x), </nowiki></center>

−

and called the least relative surprise estimator in Evans<ref name=r11>{{cite ~~res~~

+

and called the least relative surprise estimator in Evans<ref name=r11>{{cite book

| autore = Evans M

| titolo = Bayesian inference procedures derived via the concept of relative surprise

Line 420: Line 420:

| OCLC =

| related = https://scholar.google.com/scholar?q=related:wlaC1_gegT4J:scholar.google.com/&scioq=&hl=it&as_sdt=0,5

−

}}</ref>, Evans and Shakhatreh<ref name=r22>{{cite ~~res~~

+

}}</ref>, Evans and Shakhatreh<ref name=r22>{{cite book

| autore = Evans M

| autore2 = Shakhatreh M

Line 435: Line 435:

| OCLC =

| related = https://scholar.google.com/scholar?q=related:dXAPbBOLlNQJ:scholar.google.com/&scioq=&hl=it&as_sdt=0,5

−

}}</ref> and Evans and Jang<ref name=r18>{{cite ~~res~~

+

}}</ref> and Evans and Jang<ref name=r18>{{cite book

| autore = Evans M

| autore2 = Jang GH

Line 476: Line 476:

===Bias===

−

There is another issue associated with using RBΨ(ψ0 | x) to assess the evidence that ψ0 is the true value. One of the key concerns with Bayesian inference methods is that the choice of the prior can bias the analysis in various ways. An approach to dealing with the bias issue is discussed in Baskurt and Evans<ref name=r1 />. Given that the assessment of the evidence that ψ0 is true is based on RBΨ(ψ0 | x), the solution is to measure a priori whether or not the chosen prior induces bias either in favor of or against ψ0. To see how to do this, note first the Savage–Dickey ratio result (see Dickey<ref>{{cite ~~res~~

+

There is another issue associated with using RBΨ(ψ0 | x) to assess the evidence that ψ0 is the true value. One of the key concerns with Bayesian inference methods is that the choice of the prior can bias the analysis in various ways. An approach to dealing with the bias issue is discussed in Baskurt and Evans<ref name=r1 />. Given that the assessment of the evidence that ψ0 is true is based on RBΨ(ψ0 | x), the solution is to measure a priori whether or not the chosen prior induces bias either in favor of or against ψ0. To see how to do this, note first the Savage–Dickey ratio result (see Dickey<ref>{{cite book

| autore = Dickey JM

| titolo = The weighted likelihood ratio, linear hypotheses on normal location parameters

Line 504: Line 504:

−

as this is the prior probability that evidence against ψ0 will not be obtained when ψ0 is false. Note that Eq. (4) tends to decrease as ψ0' moves away from ψ0. When Eq. (4) is large, there is bias in favor of ψ0 and so subsequently reporting that evidence in favor of ψ0 being true has been found, is not convincing. For a fixed prior, both Eqs. (3) and (4) decrease with sample size and so, in design situations, they can be used to set sample size and so control bias (see Evans<ref name=self />). Considering the bias in the evidence is connected with the idea of a severe test as discussed in Popper<ref>{{cite ~~res~~

+

as this is the prior probability that evidence against ψ0 will not be obtained when ψ0 is false. Note that Eq. (4) tends to decrease as ψ0' moves away from ψ0. When Eq. (4) is large, there is bias in favor of ψ0 and so subsequently reporting that evidence in favor of ψ0 being true has been found, is not convincing. For a fixed prior, both Eqs. (3) and (4) decrease with sample size and so, in design situations, they can be used to set sample size and so control bias (see Evans<ref name=self />). Considering the bias in the evidence is connected with the idea of a severe test as discussed in Popper<ref>{{cite book

| autore = Popper KR

| titolo = The Logic of Scientific Discovery

Line 518: Line 518:

| OCLC =

| related = https://scholar.google.com/scholar?q=related:CMEFh2W1AFEJ:scholar.google.com/&scioq=&hl=it&as_sdt=0,5

−

}}</ref> and Mayo and Spanos<ref>{{cite ~~res~~

+

}}</ref> and Mayo and Spanos<ref>{{cite book

| autore = Mayo DG

| autore2 = Spanos A

Line 591: Line 591:

In general, the prosecutor's fallacy refers to any kind of error in probabilistic reasoning made by a prosecutor when arguing for the conviction of a defendant. The paper Thompson and Schumann<ref name=r30 /> seems to be one of the earliest references and so that context and its relevance to measuring statistical evidence is considered.

−

Suppose a population is split into two classes where a proportion ϵ are guilty of a crime and a proportion 1 − ϵ are not guilty. Suppose further that a particular trait is held by a proportion ψ1 of those innocent and a proportion ψ2 of those who are guilty. The overall proportion in the population possessing the trait is then (1 − ϵ)ψ1 + ϵψ2 and this will be small whenever ϵ and ψ1 are small. The values ϵ and ψ1 being small correspond to the proportion of guilty being very small and the trait being very rare in the population. The prosecutor notes that the defendant has this trait and, because (1 − ϵ)ψ1 + ϵψ2 is very small, concludes the defendant is guilty. Actually, as cited in Thompson and Schumann<ref name=r30>{{cite ~~res~~

+

Suppose a population is split into two classes where a proportion ϵ are guilty of a crime and a proportion 1 − ϵ are not guilty. Suppose further that a particular trait is held by a proportion ψ1 of those innocent and a proportion ψ2 of those who are guilty. The overall proportion in the population possessing the trait is then (1 − ϵ)ψ1 + ϵψ2 and this will be small whenever ϵ and ψ1 are small. The values ϵ and ψ1 being small correspond to the proportion of guilty being very small and the trait being very rare in the population. The prosecutor notes that the defendant has this trait and, because (1 − ϵ)ψ1 + ϵψ2 is very small, concludes the defendant is guilty. Actually, as cited in Thompson and Schumann<ref name=r30>{{cite book

| autore = Thompson WC

| autore2 = Schumann EL

Line 630: Line 630:

==Conclusions==

−

A broad outline of relative belief theory has been described here. The inferences have many nice properties like invariance under reparameterizations and a wide variety of optimal properties in the class of all Bayesian inferences. The papers Evans<ref name=r11 />, Evans, Guttman, and Swartz<ref>{{cite ~~res~~

+

A broad outline of relative belief theory has been described here. The inferences have many nice properties like invariance under reparameterizations and a wide variety of optimal properties in the class of all Bayesian inferences. The papers Evans<ref name=r11 />, Evans, Guttman, and Swartz<ref>{{cite book

| autore = Evans M

| autore2 = Guttman I

Line 646: Line 646:

| OCLC =

| related = https://scholar.google.com/scholar?q=related:_kSy5nA6oOMJ:scholar.google.com/&scioq=&hl=it&as_sdt=0,5

−

}}</ref>, Evans and Shakhatreh<ref name=r22 />, Evans and Jang<ref name=r18 /> and Baskurt and Evans<ref name=r1 /> are primarily devoted to development of the theory. Many of these papers contain applications to specific problems but also see Evans, Gilula and Guttman<ref>{{cite ~~res~~

+

}}</ref>, Evans and Shakhatreh<ref name=r22 />, Evans and Jang<ref name=r18 /> and Baskurt and Evans<ref name=r1 /> are primarily devoted to development of the theory. Many of these papers contain applications to specific problems but also see Evans, Gilula and Guttman<ref>{{cite book

| autore = Evans M

| autore2 = Gilula Z

Line 662: Line 662:

| OCLC =

| related = https://scholar.google.com/scholar?q=related:MQZI95vO1TkJ:scholar.google.com/&scioq=&hl=it&as_sdt=0,5

−

}}</ref>, Cao, Evans and Guttman<ref>{{cite ~~res~~

+

}}</ref>, Cao, Evans and Guttman<ref>{{cite book

| autore = Cao Y

| autore2 = Evans M

Line 678: Line 678:

| OCLC =

| related = https://scholar.google.com/scholar?q=related:Xcg_LUgvXJAJ:scholar.google.com/&scioq=&hl=it&as_sdt=0,5

−

}}</ref> and Muthukumarana and Evans<ref>{{cite ~~res~~

+

}}</ref> and Muthukumarana and Evans<ref>{{cite book

| autore = Muthukumarana S

| autore2 = Evans M

Gianfranco

Editor, Editors, USER, admin, Bureaucrats, Check users, dev, editor, Interface administrators, lookupuser, oversight, push-subscription-manager, Suppressors, Administrators, translator, widgeteditor

17,894

edits