The main difference is that branches are hypotheses at the track level. They are formed based on the assignment results and maintained until we prune them.
The hypotheses overall are global across all the tracks. We keep the maximum number of hypotheses that have the overall best scores and discard the rest, which means that some branches are pruned with that discarding.
Here is a simple example. Suppose you have 2 tracks from the previous step: A and B (for simplicity, let's say they each have one branch). Furthermore, let's say that MaxNumTrackBranches is 3 and MaxNumHypotheses is 5.
In the current step, we compute new branches based on the assignments and get:
Track A is split to branches A1, A2, and A3
Track B is split to branches B1, and B2.
We additionally create new tracks C with 1 branch and D with two branches.
Each one of the branches has a score and each overall hypothesis is a collection of branches and their scores. Obviously, from the example above you can see that the total number of hypotheses is 3 (for A) x 2 (for B) x 1 (for C) x 2 (for D). For example, a hypothesis is A1, B2, C, D1.
We use the maximum number of hyptoheses to keep only the top score hypotheses. Then we prune the rest and remove track branches that are not suppored by any of the surviving hypotheses. For example, let's say that A3 was a branch that didn't make it to the top MaxNumHypotheses (5) hypotheses, we would eliminate it from the list of branches we maintain, so at the end of the step, track A has only two remaining branches (A1 and A2).
This process of branching and pruning is described in the paper we refer to in the doc, if you wish to read more details.
Werthmann, J. R.. "Step-by-Step Description of a Computationally Efficient Version of Multiple Hypothesis Tracking." In International Society for Optics and Photonics, Vol. 1698, pp. 228-301, 1992.
Obviously, you would want to carefully consider how many hypotheses you would like to keep so that you're still keeping enough track branches when tracking a large number of tracks. That being said, performance and memory usage play a big role when you need to prune the branches and it is a delicate tuning and tweaking to get the right result.
I hope this helps.