This shows the results of the ultrametric tree-based, explainable, solar-powered language model
These charts update each day.
This animation uses a tiny, inspectable decision tree to predict a
next word from a sentence context. The default sentence comes from
../papers/ultratrees/ultratrees.tex.
Rule semantics are illustrative and intentionally human-readable (not the production model).
?
Training up an ultrametric tree by finding the optimal split at each step is computationally prohibitive. We can only subsample. Each order of magnitude increase in carefulness is roughly three orders of magnitude more compute time required. "Sense Annotated 1" is the alias of the first training of Careful1000, which seems like a reasonable compromise. It requires about 100 times as many nodes to achieve the same result as Careful10000, but it can train 1000 times faster.
Careful100 and Careful10 are much, much faster to train, but there's a threshold somewhere between Careful100 and Careful1000 where there are too many bad choices. It's open question what that threshold is, and why a threshold even exists.
The key question that this work set out to answer was whether sense annotation, and indeed, the whole idea of synergistic semantic and statistical models were worth exploring.
The "Unannotated Model 1" can be seen as being a baseline statistical model. It's equivalent to a one-hot encoded decision tree. The sense annotated model's learning generalises where the unannotated model is overfitting very early.
Broadly speaking, re-training on the same data yields similar results. Loss on the hold-out training data goes down, roughly linearly with the logarithm of the number of nodes in the model. Note that these are only sorted by time (the model that was trained first). It's just co-incidence that model 1 is the best and model 5 the worst.
Even the worst model is doing much better than the unannotated model. The probability of this happening by chance is 1/32, which is equivalent to a p-value of 0.03.
Ensembling works. The ensemble of 5 "Careful 1000" models gets results that don't look all that different to an extrapolation of the best of them.
Comparison with a neural baseline shows that the best-trained ultrametric trees need a few orders of magnitude more nodes than a neural network needs trainable parameters. But different ultrametric training regimes have several orders of magnitude difference within them, so it's not hard to believe that a better training regime might close this gap.
Weirder is that here sense annotation makes barely any difference to the neural network models.
This chart answers: for a given neural parameter budget, how many UltraTree nodes are needed to match the best neural total-loss result. Solid points are directly observed in evaluation data; dotted points are extrapolated because neural loss is better than the best observed UltraTree loss.
This compares estimated training compute time against parameter count
for both systems. For UltraTree, time comes from node-creation
timestamps in Postgres (ultratree.nodes.when_created) with
a 24-hour active-gap cutoff. For neural models, time is inferred from
checkpoint file modification deltas in
ultratree-neural-baseline.
Instead of looking at the total loss over all parts of speech, we would expect that nouns would get the most benefit from having sense annotation into a hierarchy.
But the data shows the exact opposite: as we train, we are increasing the loss on nouns, which means that the loss on all other parts of speech much be dropping even more rapidly.
We do see that the ultratree models soundly outperform neural network models on nouns though. Neural networks are behaving as one would expect: larger models have more generalised learning.
Theory: the ultrametric models mostly predict nouns, because nouns
are the most common part of speech in the corpus, and they can group
parts of speech together into an aggregate. The neural network mostly
predicts punctuation, since it has no way of aggregating parts of speech
together without internalising rules of grammar. The
.'' character is the most common word'' in the corpus, so
all else being equal, it will get predicted more often.
We can see which contexts get used for node splitting. (This is not the same as asking which nodes get used the most often in inference.)
These site outputs are now generated from PostgreSQL data and do not require SQLite in the build/deploy flow.
Assumptions:
raksasa as user
ultratreeULTRATREE_DATABASE_URL points to the live UltraTree
Postgres DBcd ~/ultratree
set -a
source config/ultratree.env
set +apsql "$ULTRATREE_DATABASE_URL" -Atc "
select model_type, count(*) from ultratree.evaluation_runs group by model_type order by model_type;
select count(*) from ultratree.inferences;
"This is idempotent and does not retrain models.
python3 scripts/recalculate_inference_losses_postgres.py --dry-run
python3 scripts/recalculate_inference_losses_postgres.py./scripts/build_site_from_postgres.shThis writes static output into site/dist/.
./scripts/deploy_site.sh --src site/dist --dry-run
./scripts/deploy_site.sh --src site/distcurl -I https://ultratree.symmachus.org
curl -I "https://ultratree.symmachus.org/total_loss_vs_model_size.png?bust=$(date +%s)"