A balanced review of Math Academy

And some thoughts on motivation and structure, for autodidacts

I haven’t written in a couple of months as we had another baby (kid #3, a boy) in January. So far he really likes it when we vacuum or play Metallica… not sure what that says about his personality but at least the house is clean. Also I’m still working on a major update to teachyourselfcs.com so please keep your helpful suggestions coming!

---

A number of my students rave about Math Academy, an online math program that claims extraordinary outcomes. It’s rare to find a discussion of math education on Twitter or Hacker News now without somebody chiming in to recommend it.

The reviews from thoughtful math teachers, though, are not so positive. Michael Pershan feels that it is “fundamentally broken” and can’t be fixed. Dan Meyer suggests that the learning may be illusory but that It Is Fun to Pretend That Hard Things Are Easy! Math Academy employee Justin Skycak was the most requested guest we’ve ever had for the CS Primer podcast; after his appearance, the private response from some of the educators I most respect suggested that I should have given him a harder time!

Education is complex, so it’s common for the most beloved courses and books to also be the most criticized. Structure and Interpretation of Computer Programs springs to mind, or the Dragon book. My goal here is not to resolve the conflict so much as to tease it apart, hopefully in a way that’s interesting irrespective of whether you’re in the market for a drilling app. Admittedly this will take some time (3000 words, 15 min reading time) so:

TL;DR:

Math Academy can be an effective and fun—even mildly addictive—way to drill procedural fluency. If your goal is anything more than test prep through, you should probably find a textbook or lecture series that moves you, for depth of conceptual understanding, and only use something like Math Academy as a supplement. If the Math Academy team were to just acknowledge this limitation and perhaps suggest good accompanying material, I would find it easier to recommend.

Any fool can know. The point is to understand.

— Fictional Einstein

Also a caveat: I’m concerned that by giving anything other than a rave review of Math Academy, I’ll discourage those who are finding success with it from continuing. Please, if you like Math Academy already, don’t quit!

A possibly addictive feedback loop

The core structure of a Math Academy lesson is a worked example, followed by a few typically multiple choice questions. Explanations are minimal and motivating context virtually nonexistent. The experience is roughly as follows:

MA: Let’s say you wish to differentiate an inverse reciprocal hyperbolic function

You: Ok…

MA: The derivative of arcsch(x) is […]

You: Ok I’m sure that will come in useful later

MA: Yes it’s an important node in the dependency graph of all of mathematics, you need to master it to progress. Now, here’s an example of applying the formula for a trivially different function like d/dx arcsch(3x)

You: Sure, that makes sense

MA: Follow those same steps, for these further slight variations

You: Ok…

MA: You get 10 XP! Keep going to maintain your streak! Just 100 more XP to get promoted to the next league!

The funny thing is that it does actually feel good to earn made up points and move up a leaderboard. At one point, I was racking up enough XP that a stranger on the internet noticed:

Having someone ask if I was “hooked” frankly rankled. Yes, I was enjoying the process, enough that I kept going beyond my original goal of giving the system a good try. But did I want to be hooked on this? It was fun, and the XP showed that I was (re)building procedural fluency in a topic that I do think is worthwhile. Certainly, it would be better than being hooked on some other kinds of fake internet points. But there is more to mathematics than being able to follow a sequence of rules, and I was concerned that my attention was being captured in a way that I did not in fact want.

Look where procedural fluency got me

The reason I could startle strangers on the internet with my “insane numbers” is that I’d been through this process already, 20 years ago. In high school, I studied hard for my final exams including for an advanced course which covered much of the same material as the early undergraduate level courses on Math Academy. I did well enough that I was accepted into a highly competitive program at university, intended for the most promising math students… which I promptly discovered was not me.

See, I had “earned” my place by practicing enough to score well on tests without really understanding what I was doing. Another student in the program—actually, the one who scored highest in the entrance exams—was in a similar position, and mentioned he had worked multiple times through decades of past papers, in preparation. He ended up becoming an engineer, and I ended up doing whatever it is that I do.

Meanwhile, the two people from our cohort who did become professional mathematicians were entirely different: they didn’t drill at all, but instead interrogated gaps in their understanding, found novel ways to develop their intuition, and overall focused on subtle conceptual aspects of a topic that could never conceivably be formulated as exam questions. They still did well enough on exams, but that was an afterthought.

The gap between us was confronting, and apparently insurmountable. I quietly gave up, and mostly coasted through the rest of my degree, taking as many computer science topics as I could through the math department and hoping that I would learn the lesson for next time.

Shortcomings of the diagnostic tests

Since my time at school I’ve come back to a few math topics with more of an “understand what I’m actually doing” approach, but I still have a lot of gaping wide holes, including embarrassingly in calculus. This came up recently as a barrier to my current study of biology, as I wanted to understand certain processes that are well modelled as differential equations. So in trying Math Academy, I had in mind that I might also do a better job with calculus this time.

I wasn’t surprised not to be able to test out of Calculus 1 on Math Academy, although it was telling how poorly the diagnostic captured the core issue:

• Many of the questions I answered correctly were in areas that I don’t feel I understand well, but where I suppose I had drilled enough in the past for my procedural fluency to survive 20 years of atrophy.

• Each question is timed individually, so some of my correct answers were too slow to count. These tended to be in areas where I do have some fundamental understanding, since I was able to re-derive a relationship I had forgotten, or reason about a problem creatively, albeit slowly.

• Some of the questions I got wrong were those where I felt like I understood the key concept well enough, but misunderstood something about how the question was posed, or made two thoughtless mistakes (if you make just one, they mercifully allow you to try a second problem). One slip up here cursed me with a long sequence of middle school trigonometry problems that were excruciatingly easy but couldn’t be skipped, to the point where I just retook the entire diagnostic. I will say more about the hubris of the “way” in a moment.

All of this is to say that the concept of a diagnostic test is great, and it does a fairly good—if frustrating—job of testing procedural fluency. Understandably, it does a poor job of testing other things like the depth of your conceptual understanding, or your ability to generalize. This should serve as a reminder of what you should or should not expect to get out of the platform overall.

The hubris of the “way”

Speaking of highly hyped educational products with major shortcomings, my 6yo daughter occasionally uses a program called Synthesis, and quite likes it. It claims and of course fails to be a “superhuman math tutor” but thankfully it’s just a collection of thoughtful interactive explainers and mini-games. The most common complaint I’ve heard about it is that there is not yet enough content, so they are doing ok!

Lessons are presented as conversations “with” the Synthesis “tutor”, which in practice is a few sentences of hardcoded prose followed by a multiple choice question. There is clearly a main path through the lesson, so most “choices” lead to effectively the same place at the same pace. This is not altogether terrible: I’m glad they thoughtfully designed a main lesson sequence, and the interaction mode can be cute and engaging.

That said, it is frustratingly the least “adaptive” of any math app we’ve tried. It is the only app where long sections can be both mandatory and so mind numbingly obvious that I need to speedrun it myself to keep my kid’s interest. In theory, the “interactive” structure should be more responsive than a traditional program; certainly the human teachers that some Synthesis ads condemn would allow a bored kid to progress faster.

Nothing is true about this other than the price. It’s certainly the least “adaptive” math app we’ve tried.

All it needs is a “skip lesson” button! Why is there no “skip lesson” button!?

My suspicion, having made similar mistakes in the past, is that they were too confident in the brilliance of their solution to recognize its shortcomings. A “skip lesson” button can feel like an admission of failure, and hey maybe the brilliant original idea can still be salvaged.

Math Academy fails with similar hubris, although to their credit, they do seem to be making an earnest attempt to smooth the rough edges. Allowing a second attempt at a failed diagnostic question is a good idea. But subjecting someone to literal years of unnecessary remedial “pre-requisites” is too great a punishment for a second slip up. So too is requiring review questions on a topic after any incorrect answers in a quiz: a human tutor would focus on conceptually significant misunderstandings, rather than trivial errors and typos. The justification from Math Academy for this design is likely that the program is based on “mastery”, but the kind of mastery I care about is about developing my intuition, not honing my mechanical precision.

The DAG is a lie

One of the biggest falsehoods in education is that everything we might want to learn has its tidy little place on a dependency graph. Want to be a machine learning engineer? Of course you should understand how the algorithms work, so first learn linear algebra and calculus, for which you need algebra, so of course pre-algebra, and so on. Don’t forget to do it on the systems side too! Perhaps start with CUDA, no wait computer architecture, no electrical engineering, no physics…

Look I understand the appeal. As a teacher it’s nice to be able to establish some assumed knowledge for a course so that we can build on from there. When I teach operating systems I like to assume some understanding of computer architecture so that I can explain a context switch, say, at the level of the CPU. Similarly I wouldn’t encourage you to teach calculus before arithmetic.

But should you learn integration before differentiation? I’d never considered this approach until I read David M. Bressoud’s fantastic book Calculus Reordered, which argues among other things that the conceptual basis of integration—as accumulation of thin slices—is a more intuitive starting point than tangents to a curve or rates of change. This was certainly the case for Archimedes, who computed volumes this way two millennia before Fermat and Descartes started finding tangents.

Talk to high school math teachers, and you’ll see that it’s also problematic to start with integration: problems can quickly grow too challenging algebraically, deviation from the standard sequence can leave you without much external support, and students can lose motivation when the applications move beyond areas and volumes.

Of course it’s not just differentiation vs integration. For any customary “dependency” of one topic on another, it’s quite reasonable to consider the inverse:

• If I taught operating systems before computer architecture, would that better motivate the latter?

• If someone’s excited to learn about deep learning, should we really tell them to go away and do 5 years of math first? Jeremy Howard vehemently opposes the standard bottom-up approach; his fast.ai course is extremely successful and top down.

• If my kids learnt physics intuitively at a young age, for instance in the style of Lewis Carroll Epstein’s Thinking Physics, wouldn’t this be great motivation for calculus, later?

• If data structures and algorithms were hypothetically the final course in an undergraduate computer science program rather than one of the first, wouldn’t students approach it with much more contextual understanding? And would it really be so bad for the others?

Ultimately there’s no one best way. Ideally each student would bounce between topics based on their specific background and goals, a level of flexibility that’s challenging for a school but easier in a less structured environment. If the learner is unconstrained by structure and driven to succeed, they’re served best by the meta skill of knowing when to switch focus from A to B, and when to return to A. Chess grandmasters don’t “master” openings before moving on to middle game and then endgame; all aspects of the game are studied continuously, and each informs the others.

So why do some learning resources designed for autodidacts—such as Math Academy, or the generally very good Execute Program—rely so heavily on dependency graphs? The generous answer may be that it’s unrealistic for the learner to know an appropriate ordering, and perhaps motivating for them to be shown the “ideal”. The cynical answer is that these programs take the idea of mastery learning too far, and have become dogmatic. The hubris of the DAG. After all, it’d be easy enough to present users with a suggested sequencing without strictly requiring that it be followed.

This is all part of a broader gripe I have with “mastery learning”. At first glance it all sounds well and good: yes surely we should “provide students with individualized support and repeated opportunities to demonstrate mastery” before moving on. But what happens if the student is still not passing, say due to poor curriculum design, or insignificant issues like test-taking skills, or simply boredom? Do you tell them to just try harder or give up?

I recently read Jonathan Watts’ eye opening biography of James Lovelock, a world class scientist who among other things invented the electron capture detector and first observed increased levels of CFCs in the atmosphere. Lovelock nonetheless struggled with basic arithmetic his entire life, requiring some impressive workarounds:

How would “mastery learning” have treated James Lovelock? The school system was bad enough already; thankfully he had the confidence to endure its poor judgment of him for long enough to prove himself in other ways. But if you were to place elementary school arithmetic on a dependency graph of all of mathematics, it is surely a very early vertex. If someone like Lovelock simply could not progress from there, it is a failure of the learning system, not the individual.

Social movements are good, actually

Despite these structural issues with Math Academy, it is still a great option for many learners, as evidenced by hundreds of happy users and frequent rave reviews. I wish Pershan and the other detractors had at least acknowledged this: all these students are now doing math on a regular basis, when they simply weren’t previously. What reason could we possibly have for discouraging them?

Sure, much of the positive sentiment is due to buzz. Somebody may read a Twitter thread by a Math Academy employee and test it out. They enjoy the feedback loop for a while and recommend it to others. Before long, people are recommending the program without even trying it, based on a halo of novelty and general social interest. A textbook may be much better than the corresponding Math Academy course but unlikely to generate the same level of hype.

But this is fine, even good. We are social animals, often with social motivations. It can be fun to participate in something that others are excited about, and buzz can help provide the activation energy for trying something new. Some people will stick with it for a long time, others cancel after a month, but this is simply how things are with educational products: everyone arrives with different context so no single resource will be universally ideal. If social buzz encourages a few more people to try it, great.

Understanding the glories of civilization

I’m not very socially motivated, myself. I’m glad that the hive mind suggested I try Math Academy, but social buzz and leaderboards won’t sustain my interest for long. Nor are points, streaks or other game mechanics. These may provide a little activation energy, but if I’m going to invest hundreds of hours into learning something hard, and retain what I’ve learnt a decade later, the motivation must come from elsewhere.

Every person has their own mix of motivation and the last thing I want to do is to judge another’s. But speaking personally, if a book promises to help me grapple mathematically with the chief glories of Western civilization, now THAT might sustain my interest.

This is from the introduction to Differential Equations with Applications and Historical Notes by George F. Simmons, a book as close as possible to perfect, for me, for this topic. It is not just motivating but stirring, driving, elevating. By math textbook standards, it overflows with fascination, excitement, even love.

The Simmons treatment is so vibrant that something like Math Academy sits lifeless in comparison. It feels rude to use them side by side. But that’s likely what I’ll do.

My fundamental gripe with Math Academy is that it overstates the value of procedural fluency. I wish they would conceded this, and at least suggest conceptually rich Simmons-tier books as supplements. Personally, I see the textbook or lecture series as core and Math Academy as the supplement, for extra drilling where needed, like a digital Schaum’s Outline. Resources like these are highly worthwhile but ultimately limited, so if you’re taking responsibility for the outcomes of your study effort, you should remember to mix and match as needed.