Why I Believe Linguistics Is Still Part of My Study in AI, Math, and CS

Before I start: this is a position piece. I will show facts and my opinion on how they connect, but I am not claiming any of these connections hold with certainty.

The Transfer of Structure

Like most people who grew up in China, English was my first foreign language, learned in school the ordinary way. What was not ordinary, at least for me, was what happened after. I got obsessed with going further: French, German, Spanish, and then, in a different direction entirely, Japanese and Korean.

Language Learning Path

What kept me going was not any single language. It was how much of what I had already learned carried over. When I first started learning English, what confused me most was the relationship between spelling and pronunciation, and a verb conjugation system that seemed to follow no particular logic. It was only later, after two years of French, once I had reached a B2 level, that I first noticed something: conjugation is not arbitrary. It is a remarkably regular system, and that regularity is one of the defining features of the Indo-European language family. English is, in fact, Indo-European too, but its conjugation has worn down to almost nothing next to its European relatives. Once I had French, German and Spanish suddenly became easy to pick up. Later I even dabbled in Sanskrit, and found it about as close as a living, attested language gets to the oldest, most conservative branch of the Indo-European family.

At some point I stopped being satisfied with vocabulary lists and grammar tables. I wanted to know why languages were built the way they were, not just how to use them. That curiosity is what led me to Fromkin, Rodman, and Hyams’ An Introduction to Language [1], and that book is where linguistics stopped being a tool for becoming a better polyglot and became, in my head, a science in its own right.

I was not a CS student yet. But looking back now, with a few years of computer science and mathematics behind me, I keep noticing how much of what pulled me into linguistics was already, quietly, the same thing that pulls me into AI and math today.

1. Abstraction and Taxonomy

The first thing that struck me was how language taxonomy is organized. Indo-European branches into Germanic, Romance, Slavic, and others, and each of those branches into individual languages. That is a hierarchy of abstraction, in the same sense a type hierarchy or a classification system in CS is a hierarchy of abstraction: broad categories at the top, specific instances at the bottom, and the interesting content living in what each level of the hierarchy chooses to keep and to throw away.

Language Family Taxonomy

(My own languages, plotted across their real families. Japanese and Korean sit in separate trees on purpose: they are typologically similar, both agglutinative, both SOV^{1SOV means Subject-Object-Verb, one way languages order the core parts of a sentence. Japanese and Korean are SOV; English is its more common cousin, SVO (Subject-Verb-Object): closer to “I eat rice” than to “I rice eat.”}, but they are not genetically related to each other the way French and Spanish are.)

Natural language is an emergent artifact of civilization. Nobody designed it top-down; it accreted over millennia of use. Computer languages are the opposite: deliberately engineered, versioned, specified in documents. And yet both show the same kind of internal diversity. Linguistic typology sorts languages by how they package meaning, analytic versus synthetic and agglutinative^{2Analytic languages, like Chinese, lean on word order and separate function words rather than changing the words themselves. Synthetic languages pack that information into the words instead, through endings, internal changes, or, in the agglutinative case, a chain of affixes stacked onto a stem.}, and programming paradigms do something structurally similar: imperative, functional, and logic programming are different strategies for expressing the same computations. In both cases, there is more than one way to encode the same expressive power, and the differences between the ways are not arbitrary, they are trade-offs.

There is a deeper version of this parallel that I find harder to let go of. Language is, in some sense, a low-dimensional projection of the world we live in. A handful of phonemes^{3A phoneme is the smallest unit of sound that changes meaning in a language. The difference between bat and pat is exactly one phoneme.}, a finite vocabulary, a finite set of grammatical rules. And yet, in another sense, it is more expressive than the world itself. Linguists have a name for this: Hockett called it displacement and productivity^{4Displacement is the ability to refer to things not present here and now: the past, the future, places never visited, things that do not exist. Productivity is the ability to produce and understand sentences that have never been said before, using a finite set of rules.}, two of the design features that set human language apart [2]. Humboldt called the underlying idea an infinite use of finite means [3], a phrase Chomsky later picked up and put at the center of generative grammar [4]. A grammar this compressed should not be able to say this much, and yet it does. That is exactly the property the next section explains mechanically, through recursion.

Whether this is the reason human intelligence differs from other animals, or just one piece of a much larger picture that includes shared intentionality and cumulative culture, is a genuinely open question in cognitive science. I lean toward thinking it matters a great deal. I am aware that is a position, not a settled fact.

2. Shared Machinery: Syntax Trees

Once I started drawing syntax trees for natural-language sentences, the resemblance to something else I had seen became impossible to ignore.

Syntax vs Abstract Syntax Tree

The tree on the left is a phrase-structure analysis of the cat sat on the mat, in the generative-grammar tradition that traces back to Chomsky [5]. The tree on the right is the abstract syntax tree a compiler would build for x = a + b * c. Different alphabets, different rules, the same shape. Both are recursive: a phrase can contain a phrase, an expression can contain an expression, and the recursion is exactly what lets a small, finite grammar generate an unbounded set of well-formed sentences or programs.

This is not just a loose resemblance. Chomsky’s own early work classified grammars by how much generative power their rules carry, a ladder now called the Chomsky hierarchy, running from simple regular grammars up to unrestricted ones [5]. Programming-language syntax and a large slice of natural-language syntax both sit at the same rung of that ladder.

DefinitionContext-Free Grammar

A context-free grammar is a 4-tuple $G = (N, \Sigma, R, S)$ , where $N$ is a finite set of nonterminal symbols, $\Sigma$ is a finite set of terminal symbols disjoint from $N$ , $R$ is a finite set of production rules of the form $A \to \alpha$ with $A \in N$ and $\alpha \in (N \cup \Sigma)^*$ , and $S \in N$ is the start symbol. A string is generated by repeatedly rewriting a nonterminal according to some rule in $R$ , beginning from $S$ , until no nonterminals remain. Because each rule rewrites a single symbol independently of its surrounding context, rules may be applied inside one another without limit, which is exactly what allows a finite $R$ to generate an unbounded language.

A parser for a programming language and a parser for a natural-language sentence are, formally, recognizing strings generated by the same class of grammar.

Underneath the label “linguistics” and the label “computer science,” what is actually doing the work here is recursion and hierarchical structure. That is a mathematical notion first, and only secondarily a linguistic or computational one.

3. The Algebra Underneath

I want to be upfront here: I am not trained in this area, and what follows is closer to a direction I want to explore than something I can defend in depth. But it is the thread that pulls this whole essay furthest toward pure math, so it earns a place here even in rough form.

Combinatory Categorial Grammar assigns each word a syntactic type, and derives sentence structure from how adjacent types combine [6].

DefinitionCombinatory Categorial Grammar

The set of categories is defined recursively: a finite set of atomic categories (for instance $S$ , $N$ , $NP$ , $PP$ ) are categories, and if $X$ and $Y$ are categories, then so are the complex categories $X/Y$ and $X\backslash Y$ . Adjacent categories combine by two rules:

X/Y \;\; Y \Rightarrow X \qquad (\text{forward application, } >)

Y \;\; X\backslash Y \Rightarrow X \qquad (\text{backward application, } <)

A transitive verb such as sat is assigned the category $(S\backslash NP)/PP$ : forward application with a $PP$ to its right yields $S\backslash NP$ , and backward application of that result with an $NP$ to its left yields the sentence $S$ .

This is not a metaphor for algebra: it is algebra, since combination is literally function application on types, and its mathematical root goes back to Joachim Lambek, whose Lambek calculus modeled the categories as a partially ordered residuated monoid, with $/$ and $\backslash$ as the residuals of concatenation [7].

RemarkA Personal Rediscovery

I did not learn about categorial grammar and lambda calculus in that order by design. I came across CCG’s function-application view of syntax through linguistics reading, well before I had any formal computer science training. The resemblance to lambda calculus only became obvious later, in Monash University’s FIT2102, Programming Paradigms, where lambda calculus is taught as the mathematical core underneath functional programming. Two courses, two departments, no shared reading list, and yet the same operation, application, kept showing up under different names.

From there the thread keeps going somewhere I find genuinely exciting: the Curry-Howard correspondence [8].

DefinitionCurry-Howard Correspondence

For intuitionistic propositional logic paired with the simply typed lambda calculus, there is a structure-preserving correspondence between the two systems: propositions correspond to types, proofs of a proposition correspond to terms of the corresponding type, and the normalization of a proof, the elimination of detours, corresponds to $\beta$ -reduction of the corresponding term. Constructing a proof and writing a well-typed program are, under this correspondence, the same activity in two vocabularies.

This ties logic, computation, and, through categorial grammar, language itself into one algebraic picture. Montague grammar supplies the semantic half of the same story, treating sentence meaning as a function built up compositionally via the lambda calculus [9], [10]. I do not yet have the formal training to argue this precisely, but I am confident enough in the outline to call it the most concrete bridge from linguistics to pure mathematics that I have found so far, and reason enough to keep pulling on it.

The Complication: When Linguists Got Fired From AI

None of this history is a straight line toward vindication. For most of the twentieth century, computational approaches to language were built by linguists, hand-writing grammars and rules. Then, starting in the late 1980s, statistical methods trained on data began outperforming them, first in speech recognition and machine translation, later almost everywhere. As recounted by Jurafsky and Martin [11], Frederick Jelinek is said to have remarked that every time he fired a linguist, the performance of his recognizer went up. It is folklore more than a citation, the exact origin of the line is genuinely murky, but the shift it describes was real, and it kept going: today’s neural models often learn directly from raw data, without being told any explicit grammatical structure at all.

It would be a clean story if it ended there: structure lost, statistics won. It does not end there. Work probing what BERT and other transformer models actually represent internally has found that they implicitly encode something that looks a great deal like syntax-tree structure, without ever being told to build one [12], [13], [14]. The structure did not disappear. It moved from being hand-engineered to being rediscovered, implicitly, as a side effect of learning to predict text well.

There is an older, harder version of this same disagreement that has never fully closed. Fodor and Pylyshyn argued that connectionist models fundamentally lack true compositionality^{5Compositionality is the idea that the meaning of a whole expression is built systematically from the meanings of its parts and how they are combined, rather than memorized whole.}, that they can approximate structured behavior without actually representing structure the way a symbolic system does [15]. More recent work on systematic generalization asks essentially the same question of today’s sequence models: do they generalize the way a compositional system would, or only in ways that happen to look similar on the data they were trained on [16]. I do not think this question is settled. I am not sure it can be settled from where we currently stand.

Closing: The Common Notion

Taxonomy as abstraction, syntax trees shared between linguistics and compilers, categorial grammar as algebra, the long argument between symbolic structure and statistical learning: I keep finding the same idea underneath all of them. Modeling structure through pattern is a mathematical notion, and both linguistics and computer science are applied fields sitting on top of it, not the source of it.

That is also, I think, why the low-dimensional-yet-unbounded property of language from the first section is not just a nice turn of phrase. It names the same question representation learning asks of a model today: how a finite, low-dimensional representation can still support a combinatorially large, often unbounded, space of distinct meanings. Continuous approximations of discrete, unbounded compositionality are exactly what today’s dense vector models are attempting to achieve. That is not a coincidence to me. It is the same curiosity, pointed at a different substrate, and it is a large part of why linguistics never actually left my study of AI and math, it just went quiet for a while.

As I said at the start: these are parallels I find worth sitting with, not a claim that linguistics literally is computer science, or math. I am not sure yet how far the analogy goes. I intend to keep pulling on it.

References

[1] V. Fromkin, R. Rodman, and N. Hyams, An Introduction to Language, 11th ed. Cengage Learning, 2017. ↩
[2] C. F. Hockett, “The Origin of Speech,” Scientific American, vol. 203, no. 3, pp. 88–96, 1960. ↩
[3] W. von Humboldt, On Language: The Diversity of Human Language-Structure and Its Influence on the Mental Development of Mankind. Cambridge University Press, 1988. Translated by Peter Heath. Originally published as Über die Verschiedenheit des menschlichen Sprachbaues, Berlin, 1836.. ↩
[4] N. Chomsky, Aspects of the Theory of Syntax. MIT Press, 1965. The phrase "infinite use of finite means," credited to Humboldt, appears on p. 8.. ↩
[5] N. Chomsky, Syntactic Structures. Mouton, 1957. a b
[6] M. Steedman, The Syntactic Process. MIT Press, 2000. ↩
[7] J. Lambek, “The Mathematics of Sentence Structure,” The American Mathematical Monthly, vol. 65, no. 3, pp. 154–170, 1958. https://doi.org/10.2307/2310058 ↩
[8] W. A. Howard, “The Formulae-as-Types Notion of Construction,” To H.B. Curry: Essays on Combinatory Logic, Lambda Calculus and Formalism, pp. 479–490, 1980. Circulated as an unpublished manuscript since 1969.. https://www.cs.cmu.edu/~crary/819-f09/Howard80.pdf ↩
[9] R. Montague, “English as a Formal Language,” Formal Philosophy: Selected Papers of Richard Montague, pp. 188–221, 1974. Originally published 1970, in Linguaggi nella Società e nella Tecnica, Edizioni di Comunità, Milan.. ↩
[10] R. Montague, “The Proper Treatment of Quantification in Ordinary English,” Approaches to Natural Language, pp. 221–242, 1973. ↩
[11] D. Jurafsky and J. H. Martin, Speech and Language Processing, 2nd ed. Prentice Hall, 2009. Recounts Frederick Jelinek's remark, often quoted as "every time I fire a linguist, the performance of the recognizer goes up" (p. 83, may shift slightly by printing).. ↩
[12] J. Hewitt and C. D. Manning, “A Structural Probe for Finding Syntax in Word Representations,” Proceedings of NAACL-HLT 2019, pp. 4129–4138, 2019. https://aclanthology.org/N19-1419 ↩
[13] I. Tenney, D. Das, and E. Pavlick, “BERT Rediscovers the Classical NLP Pipeline,” arXiv:1905.05950, 2019. ACL 2019, pp. 4593--4601.. https://arxiv.org/abs/1905.05950 ↩
[14] G. Jawahar, B. Sagot, and D. Seddah, “What Does BERT Learn about the Structure of Language?,” Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019. https://aclanthology.org/P19-1356 ↩
[15] J. A. Fodor and Z. W. Pylyshyn, “Connectionism and Cognitive Architecture: A Critical Analysis,” Cognition, vol. 28, no. 1-2, pp. 3–71, 1988. ↩
[16] B. M. Lake and M. Baroni, “Generalization without Systematicity: On the Compositional Skills of Sequence-to-Sequence Recurrent Networks,” arXiv:1711.00350, 2018. ICML 2018.. https://arxiv.org/abs/1711.00350 ↩

SOV means Subject-Object-Verb, one way languages order the core parts of a sentence. Japanese and Korean are SOV; English is its more common cousin, SVO (Subject-Verb-Object): closer to “I eat rice” than to “I rice eat.” ↩
Analytic languages, like Chinese, lean on word order and separate function words rather than changing the words themselves. Synthetic languages pack that information into the words instead, through endings, internal changes, or, in the agglutinative case, a chain of affixes stacked onto a stem. ↩
A phoneme is the smallest unit of sound that changes meaning in a language. The difference between bat and pat is exactly one phoneme. ↩
Displacement is the ability to refer to things not present here and now: the past, the future, places never visited, things that do not exist. Productivity is the ability to produce and understand sentences that have never been said before, using a finite set of rules. ↩
Compositionality is the idea that the meaning of a whole expression is built systematically from the meanings of its parts and how they are combined, rather than memorized whole. ↩