Abstract
Criticality has been regarded as a hallmark of complex, sophisticated systems and has been studied extensively in the sciences. In this thesis, we study critical changes in information produced by natural systems over various variables, in an effort to understand the relationships between these variables and the internal state of these systems as well as their complexity. Though typically viewed from the lens of power laws, we use the fact that critical points produce large changes in system behavior to devise a method to find the critical points in a system. We use an information-theoretic distance such as the Kullback- Leibler divergence between observations at consecutive values of an independent variable as a measure, which we refer to as divergence rate, Mt, and a local optimizer to find the peaks in this measure. These peaks indicate large changes in system behaviour and hence a critical point. We find that most systems exhibit a finite number of critical points which correspond to boundaries between levels of information compression in the representation of the system. Furthermore, by utilizing a maximum entropy objective function which scales tractably to higher orders, we extend maximum entropy models to higher orders for the first time, improving several theoretical properties of such models, including probability density estimation. We have found that using this method to model discrete natural systems allows the determination of two important types of complexities in systems: the order of interaction and sparsity structure. Such an understanding of system complexity and structure will allow us to more efficiently use resources to harness the principles of their operation.