Bayesian vs. Frequentist analysis

Reasons besides the use of subjective priors why Bayesian and Frequentist approaches are different:

"There is a popular myth that states that Bayesian methods differ from orthodox (also known as “frequentist” or “sampling theory”) statistical methods only by the inclusion of subjective priors that are arbitrary and difficult to assign, and usually do not make much difference to the conclusions. It is true that at the first level of
inference, a Bayesian’s results will often differ little from the outcome of an orthodox attack. What is not widely appreciated is how Bayes performs
the second level of inference. It is here that Bayesian methods are totally different from orthodox methods. Indeed, when regression and density estimation are discussed in most statistics texts, the task of model comparison is virtually ignored; no general orthodox method exists for solving this problem.

Model comparison is a difficult task because it is not possible simply to choose the model that fits the data best: more complex models can always fit the data better, so the maximum likelihood model choice would lead us inevitably to implausible overparameterized models that generalize poorly. “Occam’s razor” is the principle that states that unnecessarily complex models should not be preferred to simpler ones. Bayesian methods automatically and quantitatively embody Occam’s razor (Gull 1988; Jeffreys 19391, without the introduction of ad hoc penalty terms. Complex models are automatically self-penalizing under Bayes’ rule."

MacKay-Bayesian Interpolation