

===== PAGE 2 =====
0.9
0.8 0.9 0.1 0.90.4 0.8 0.1 0.90.1 0.8 0.1
0.4 0.1
(a) detection boxes
(b) trackletsby associating high score detection boxes
(c) trackletsby associating every detection box
Framet1 Frame t2 Frame t3
Figure 2. Examples of our method which associates every detec-
tion box. (a) shows all the detection boxes with their scores. (b)
shows the tracklets obtained by previous methods which associates
detection boxes whose scores are higher than a threshold, i.e. 0.5.
The same box color represents the same identity. (c) shows the
tracklets obtained by our method. The dashed boxes represent the
predicted box of the previous tracklets using Kalman Filter. The
two low score detection boxes are correctly matched to the previ-
ous tracklets based on the large IoU.
0.1. These detection boxes are eliminated by the threshold-
ing mechanism and the red tracklet disappears accordingly.
Nevertheless, if we take every detection box into consider-
ation, more false positives will be introduced immediately,
e.g., the most right box in frame t3 of Figure 2 (a). To the
best of our knowledge, very few methods [30, 63] in MOT
are able to handle this detection dilemma.
In this paper, we identify that the similarity with tracklets
provides a strong cue to distinguish the objects and back-
ground in low score detection boxes. As shown in Figure 2
(c), two low score detection boxes are matched to the track-
lets by the motion model’s predicted boxes, and thus the
objects are correctly recovered. At the same time, the back-
ground box is removed since it has no matched tracklet.
For making full use of detection boxes from high scores
to low ones in the matching process, we present a simple
and effective association method BYTE, named for each de-
tection box is a basic unit of the tracklet, as byte in computer
program, and our tracking method values every detailed de-
tection box. We ﬁrst match the high score detection boxes
to the tracklets based on motion similarity or appearance
similarity. Similar to [6], we adopt Kalman ﬁlter [29] to
predict the location of the tracklets in the new frame. The
similarity can be computed by the IoU or Re-ID feature dis-
tance of the predicted box and the detection box. Figure 2
(b) is exactly the results after the ﬁrst matching. Then, we
perform the second matching between the unmatched track-
lets, i.e. the tracklet in red box, and the low score detection
boxes using the same motion similarity. Figure 2 (c) shows
the results after the second matching. The occluded person
with low detection scores is matched correctly to the pre-
vious tracklet and the background (in the right part of the
image) is removed.
As the integrating topic of object detection and asso-
ciation, a desirable solution to MOT is never a detector
and the following association; besides, well-designed of
their junction area is also important. The innovation of
BYTE lies in the junction area of detection and associa-
tion, where low score detection boxes are bridges to boost
both of them. Beneﬁting from this integration innovation,
when BYTE is applied to 9 different state-of-the-art track-
ers, including the Re-ID-based ones [33,47,69,85], motion-
based ones [71, 89], chain-based one [48] and attention-
based ones [59, 80], notable improvements are achieved on
almost all the metrics including MOTA, IDF1 score and ID
switches. For example, we increase the MOTA of Center-
Track [89] from 66.1 to 67.4, IDF1 from 64.2 to 74.0 and
decrease the IDs from 528 to 144 on the half validation set
of MOT17.
Towards pushing forwards the state-of-the-art perfor-
mance of MOT, we propose a simple and strong tracker,
named ByteTrack. We adopt a recent high-performance
detector YOLOX [24] to obtain the detection boxes and
associate them with our proposed BYTE. On the MOT
challenges, ByteTrack ranks 1st on both MOT17 [44] and
MOT20 [17], achieving 80.3 MOTA, 77.3 IDF1 and 63.1
HOTA with 30 FPS running speed on V100 GPU on
MOT17 and 77.8 MOTA, 75.2 IDF1 and 61.3 HOTA on
much more crowded MOT20. ByteTrack also achieves
state-of-the-art performance on HiEve [37] and BDD100K
[79] tracking benchmarks. We hope the efﬁciency and sim-
plicity of ByteTrack could make it attractive in real applica-
tions such as social computing.
2. Related Work
2.1. Object Detection in MOT
Object detection is one of the most active topics in
computer vision and it is the basis of multi-object track-
ing. The MOT17 dataset [44] provides detection results
obtained by popular detectors such as DPM [22], Faster
R-CNN [50] and SDP [77]. A large number of methods
[3,9,12,14,28,74,91] focus on improving the tracking per-
formance based on these given detection results.


===== PAGE 3 =====
Tracking by detection. With the rapid development of ob-
ject detection [10, 23, 26, 35, 49, 50, 58, 60], more and more
methods begin to utilize more powerful detectors to obtain
higher tracking performance. The one-stage object detec-
tor RetinaNet [35] begin to be adopted by several methods
such as [39, 48]. CenterNet [90] is the most popular de-
tector adopted by most methods [63, 65, 67, 71, 85, 87, 89]
for its simplicity and efﬁciency. The YOLO series detec-
tors [8, 49] are also adopted by a large number of meth-
ods [15, 33, 34, 69] for its excellent balance of accuracy and
speed. Most of these methods directly use the detection
boxes on a single image for tracking.
However, the number of missing detections and very low
scoring detections begin to increase when occlusion or mo-
tion blur happens in the video sequence, as is pointed out
by video object detection methods [41, 62]. Therefore, the
information of the previous frames are usually leveraged to
enhance the video detection performance.
Detection by tracking. Tracking can also adopted to
help obtain more accurate detection boxes. Some meth-
ods [12–15, 53, 91] utilize single object tracking (SOT) [5]
or Kalman ﬁlter [29] to predict the location of the tracklets
in the following frame and fuse the predicted boxes with
the detection boxes to enhance the detection results. Other
methods [34, 86] leverage tracked boxes in the previous
frames to enhance feature representation of the following
frame. Recently, Transformer-based [20, 38, 64, 66] detec-
tors [11, 92] are adopted by several methods [42, 59, 80] for
its strong ability to propagate boxes between frames. Our
method also utilize the similarity with tracklets to strength
the reliability of detection boxes.
After obtaining the detection boxes by various detectors,
most MOT methods [33, 39, 47, 59, 69, 71, 85] only keep
the high score detection boxes by a threshold, i.e. 0.5, and
use those boxes as the input of data association. This is
because the low score detection boxes contain many back-
grounds which harm the tracking performance. However,
we observe that many occluded objects can be correctly de-
tected but have low scores. To reduce missing detections
and keep the persistence of trajectories, we keep all the de-
tection boxes and associate across every of them.
2.2. Data Association
Data association is the core of multi-object tracking,
which ﬁrst computes the similarity between tracklets and
detection boxes and leverage different strategies to match
them according to the similarity.
Similarity metrics. Location, motion and appearance are
useful cues for association. SORT [6] combines location
and motion cues in a very simple way. It ﬁrst adopts Kalman
ﬁlter [29] to predict the location of the tracklets in the new
frame and then computes the IoU between the detection
boxes and the predicted boxes as the similarity. Some re-
cent methods [59, 71, 89] design networks to learn object
motions and achieve more robust results in cases of large
camera motion or low frame rate. Location and motion
similarity are accurate in the short-range matching. Ap-
pearance similarity are helpful in the long-range matching.
An object can be re-identiﬁed using appearance similarity
after being occluded for a long period of time. Appear-
ance similarity can be measured by the cosine similarity of
the Re-ID features. DeepSORT [70] adopts a stand-alone
Re-ID model to extract appearance features from the de-
tection boxes. Recently, joint detection and Re-ID mod-
els [33, 39, 47, 69, 84, 85] becomes more and more popular
because of their simplicity and efﬁciency.
Matching strategy. After similarity computation, match-
ing strategy assigns identities to the objects. This can be
done by Hungarian Algorithm [31] or greedy assignment
[89]. SORT [6] matches the detection boxes to the track-
lets by once matching. DeepSORT [70] proposes a cas-
caded matching strategy which ﬁrst matches the detection
boxes to the most recent tracklets and then to the lost ones.
MOTDT [12] ﬁrst utilizes appearance similarity to match
and then utilize the IoU similarity to match the unmatched
tracklets. QDTrack [47] turns the appearance similarity into
probability by a bi-directional softmax operation and adopts
a nearest neighbor search to accomplish matching. Atten-
tion mechanism [64] can directly propagate boxes between
frames and perform association implicitly. Recent methods
such as [42,80] propose track queries to ﬁnd the location of
the tracked objects in the following frames. The matching
is implicitly performed in the attention interaction process
without using Hungarian Algorithm.
All these methods focus on how to design better associ-
ation methods. However, we argue that the way detection
boxes are utilized determines the upper bound of data asso-
ciation and we focus on how to make full use of detection
boxes from high scores to low ones in the matching process.
3. BYTE
We propose a simple, effective and generic data asso-
ciation method, BYTE. Different from previous methods
[33, 47, 69, 85] which only keep the high score detection
boxes, we keep almost every detection box and separate
them into high score ones and low score ones. We ﬁrst asso-
ciate the high score detection boxes to the tracklets. Some
tracklets get unmatched because they do not match to an ap-
propriate high score detection box, which usually happens
when occlusion, motion blur or size changing occurs. We
then associate the low score detection boxes and these un-
matched tracklets to recover the objects in low score detec-
tion boxes and ﬁlter out background, simultaneously. The


===== PAGE 4 =====
Algorithm 1: Pseudo-code of BYTE.
Input: A video sequence V; object detector Det; detection score
thresholdτ
Output: TracksT of the video
1 Initialization:T ←∅
2 for framefk in V do
/* Figure 2(a) */
/* predict detection boxes & scores */
3 Dk← Det(fk)
4 Dhigh←∅
5 Dlow←∅
6 ford inDk do
7 ifd.score>τ then
8 Dhigh←D high∪{d}
9 end
10 else
11 Dlow←D low∪{d}
12 end
13 end
/* predict new locations of tracks */
14 fort inT do
15 t← KalmanFilter(t)
16 end
/* Figure 2(b) */
/* first association */
17 AssociateT andDhigh using Similarity#1
18 Dremain← remaining object boxes fromDhigh
19 Tremain← remaining tracks fromT
/* Figure 2(c) */
/* second association */
20 AssociateTremain andDlow using similarity#2
21 Tre−remain← remaining tracks fromTremain
/* delete unmatched tracks */
22 T ←T\T re−remain
/* initialize new tracks */
23 ford inDremain do
24 T ←T∪{ d}
25 end
26 end
27 Return:T
Track rebirth [70,89] is not shown in the algorithm for simplicity. In green
is the key of our method.
pseudo-code of BYTE is shown in Algorithm 1.
The input of BYTE is a video sequence V, along with an
object detector Det. We also set a detection score threshold
τ. The output of BYTE is the tracksT of the video and each
track contains the bounding box and identity of the object
in each frame.
For each frame in the video, we predict the detection
boxes and scores using the detector Det. We separate all
the detection boxes into two partsDhigh andDlow accord-
ing to the detection score threshold τ. For the detection
boxes whose scores are higher thanτ, we put them into the
high score detection boxesDhigh. For the detection boxes
whose scores are lower than τ, we put them into the low
score detection boxesDlow (line 3 to 13 in Algorithm 1).
After separating the low score detection boxes and the
high score detection boxes, we adopt Kalman ﬁlter to pre-
dict the new locations in the current frame of each track in
T (line 14 to 16 in Algorithm 1).
The ﬁrst association is performed between the high score
detection boxesDhigh and all the tracks T (including the
lost tracksTlost). Similarity#1 can be computed by
either by the IoU or the Re-ID feature distances between
the detection boxesDhigh and the predicted box of tracks
T . Then, we adopt Hungarian Algorithm [31] to ﬁnish the
matching based on the similarity. We keep the unmatched
detections inDremain and the unmatched tracks inTremain
(line 17 to 19 in Algorithm 1).
BYTE is highly ﬂexible and can be compatible to other
different association methods. For example, when BYTE is
combined with FairMOT [85], Re-ID feature is added into
* first association * in Algorithm 1, others are
the same. In the experiments, we apply BYTE to 9 different
state-of-the-art trackers and achieve notable improvements
on almost all the metrics.
The second association is performed between the low
score detection boxes Dlow and the remaining tracks
Tremain after the ﬁrst association. We keep the unmatched
tracks inTre−remain and just delete all the unmatched low
score detection boxes, since we view them as background.
(line 20 to 21 in Algorithm 1). We ﬁnd it important to use
IoU alone as the Similarity#2 in the second associa-
tion because the low score detection boxes usually contains
severe occlusion or motion blur and appearance features are
not reliable. Thus, when apply BYTE to other Re-ID based
trackers [47, 69, 85], we do not adopt appearance similarity
in the second association.
After the association, the unmatched tracks will be
deleted from the tracklets. We do not list the procedure of
track rebirth [12, 70, 89] in Algorithm 1 for simplicity. Ac-
tually, it is necessary for the long-range association to pre-
serve the identity of the tracks. For the unmatched tracks
Tre−remain after the second association, we put them into
Tlost. For each track in Tlost, only when it exists for more
than a certain number of frames, i.e. 30, we delete it from
the tracksT . Otherwise, we remain the lost tracks Tlost in
T (line 22 in Algorithm 1).Finally, we initialize new tracks
from the unmatched high score detection boxesDremain af-
ter the ﬁrst association. (line 23 to 27 in Algorithm 1).The
output of each individual frame is the bounding boxes and
identities of the tracksT in the current frame. Note that we
do not output the boxes and identities ofTlost.
To put forwards the state-of-the-art performance of
MOT, we design a simple and strong tracker, named
ByteTrack, by equipping the high-performance detector
YOLOX [24] with our association method BYTE.


===== PAGE 5 =====
4. Experiments
4.1. Setting
Datasets. We evaluate BYTE and ByteTrack on MOT17
[44] and MOT20 [17] datasets under the “private detection”
protocol. Both datasets contain training sets and test sets,
without validation sets. For ablation studies, we use the ﬁrst
half of each video in the training set of MOT17 for training
and the last half for validation following [89]. We train on
the combination of CrowdHuman dataset [55] and MOT17
half training set following [59, 71, 80, 89]. We add Cityper-
son [82] and ETHZ [21] for training following [33, 69, 85]
when testing on the test set of MOT17. We also test Byte-
Track on HiEve [37] and BDD100K [79] datasets. HiEve
is a large scale human-centric dataset focusing on crowded
and complex events. BDD100K is the largest driving video
dataset and the dataset splits of the MOT task are 1400
videos for training, 200 videos for validation and 400 videos
for testing. It needs to track objects of 8 classes and contains
cases of large camera motion.
Metrics. We use the CLEAR metrics [4], including MOTA,
FP, FN, IDs,etc., IDF1 [51] and HOTA [40] to evaluate dif-
ferent aspects of the tracking performance. MOTA is com-
puted based on FP, FN and IDs. Considering the amount
of FP and FN are larger than IDs, MOTA focuses more
on the detection performance. IDF1 evaluates the identity
preservation ability and focus more on the association per-
formance. HOTA is a very recently proposed metric which
explicitly balances the effect of performing accurate detec-
tion, association and localization. For BDD100K dataset,
there are some multi-class metrics such as mMOTA and
mIDF1. mMOTA / mIDF1 is computed by averaging the
MOTA / IDF1 of all the classes.
Implementation details. For BYTE, the default detection
score threshold τ is 0.6, unless otherwise speciﬁed. For
the benchmark evaluation of MOT17, MOT20 and HiEve,
we only use IoU as the similarity metrics. In the linear as-
signment step, if the IoU between the detection box and
the tracklet box is smaller than 0.2, the matching will be re-
jected. For the lost tracklets, we keep it for 30 frames in case
it appears again. For BDD100K, we use UniTrack [68] as
the Re-ID model. In ablation study, we use FastReID [27]
to extract Re-ID features for MOT17.
For ByteTrack, the detector is YOLOX [24] with
YOLOX-X as the backbone and COCO-pretrained model
[36] as the initialized weights. For MOT17, the train-
ing schedule is 80 epochs on the combination of MOT17,
CrowdHuman, Cityperson and ETHZ. For MOT20 and
HiEve, we only add CrowdHuman as additional training
data. For BDD100K, we do not use additional training data
and only train 50 epochs. The input image size is 1440
×800 and the shortest side ranges from 576 to 1024 during
multi-scale training. The data augmentation includes Mo-
saic [8] and Mixup [81]. The model is trained on 8 NVIDIA
Tesla V100 GPU with batch size of 48. The optimizer is
SGD with weight decay of 5× 10−4 and momentum of
0.9. The initial learning rate is 10−3 with 1 epoch warm-
up and cosine annealing schedule. The total training time
is about 12 hours. Following [24], FPS is measured with
FP16-precision [43] and batch size of 1 on a single GPU.
4.2. Ablation Studies on BYTE
Similarity analysis. We choose different types of sim-
ilarity for the ﬁrst association and the second association
of BYTE. The results are shown in Table 1. We can
see that either IoU or Re-ID can be a good choice for
Similarity#1 on MOT17. IoU achieves better MOTA
and IDs while Re-ID achieves higher IDF1. On BDD100K,
Re-ID achieves much better results than IoU in the ﬁrst as-
sociation. This is because BDD100K contains large camera
motion and the annotations are in low frame rate, which
causes failure of motion cues. It is important to utilize
IoU as Similarity#2 in the second association on both
datasets because the low score detection boxes usually con-
tains severe occlusion or motion blur and thus Re-ID fea-
tures are not reliable. From Table 1 we can ﬁnd that using
IoU as Similarity#2 increases about 1.0 MOTA com-
pared to Re-ID, which indicates that Re-ID features of the
low score detection boxes are not reliable.
Comparisons with other association methods. We com-
pare BYTE with other popular association methods includ-
ing SORT [6], DeepSORT [70] and MOTDT [12] on the
validation set of MOT17 and BDD100K. The results are
shown in Table 2.
SORT can be seen as our baseline method because both
methods only adopt Kalman ﬁlter to predict the object mo-
tion. We can ﬁnd that BYTE improves the MOTA metric
of SORT from 74.6 to 76.6, IDF1 from 76.9 to 79.3 and
decreases IDs from 291 to 159. This highlights the impor-
tance of the low score detection boxes and proves the ability
of BYTE to recover object boxes from low score one.
DeepSORT utilizes additional Re-ID models to enhance
the long-range association. We surprisingly ﬁnd BYTE also
has additional gains compared with DeepSORT. This sug-
gests a simple Kalman ﬁlter can perform long-range asso-
ciation and achieve better IDF1 and IDs when the detection
boxes are accurate enough. We note that in severe occlusion
cases, Re-ID features are vulnerable and may lead to iden-
tity switches, instead, motion model behaves more reliably.
MOTDT integrates motion-guided box propagation re-
sults along with detection results to associate unreliable de-
tection results with tracklets. Although sharing the simi-


===== PAGE 6 =====
MOT17 BDD100K
Similarity#1 Similarity#2 MOTA ↑ IDF1↑ IDs↓ mMOTA↑ mIDF1↑ IDs↓
IoU Re-ID 75.8 77.5 231 39.2 48.3 29172
IoU IoU 76.6 79.3 159 39.4 48.9 27902
Re-ID Re-ID 75.2 78.7 276 45.0 53.4 10425
Re-ID IoU 76.3 80.5 216 45.5 54.8 9140
Table 1. Comparison of different type of similarity metrics used in the ﬁrst association and the second association of BYTE on MOT17
and BDD100K validation set. The best results are shown in bold.
MOT17 BDD100K
Method w/ Re-ID MOTA ↑ IDF1↑ IDs↓ mMOTA↑ mIDF1↑ IDs↓ FPS
SORT 74.6 76.9 291 30.9 41.3 10067 30.1
DeepSORT ✓ 75.4 77.2 239 24.5 38.2 10720 13.5
MOTDT ✓ 75.8 77.6 273 26.7 39.8 14520 11.1
BYTE (ours) 76.6 79.3 159 39.4 48.9 27902 29.6
BYTE (ours) ✓ 76.3 80.5 216 45.5 54.8 9140 11.8
Table 2. Comparison of different data association methods on MOT17 and BDD100K validation set. The best results are shown in bold.
Figure 3. Comparison of the performances of BYTE and SORT
under different detection score thresholds. The results are from
the validation set of MOT17.
lar motivation, MOTDT is behind BYTE by a large mar-
gin. We explain that MOTDT uses propagated boxes as
tracklet boxes, which may lead to locating drifts in track-
ing. Instead, BYTE uses low-score detection boxes to
re-associate those unmatched tracklets, therefore, tracklet
boxes are more accurate.
Table 2 also shows the results on BDD100K dataset.
BYTE also outperforms other association methods by a
large margin. Kalman ﬁlter fails in autonomous driving
scenes and it is the main reason for the low performance
of SORT, DeepSORT and MOTDT. Thus, we do not use
Kalman ﬁlter on BDD100K. Additional off-the-shelf Re-
ID models greatly improve the performance of BYTE on
BDD100K.
Robustness to detection score threshold. The detection
score threshold τhigh is a sensitive hyper-parameter and
needs to be carefully tuned in the task of multi-object track-
ing. We change it from 0.2 to 0.8 and compare the MOTA
Figure 4. Comparison of the number of TPs and FPs in all low
score detection boxes and the low score tracked boxes obtained by
BYTE. The results are from the validation set of MOT17.
and IDF1 score of BYTE and SORT. The results are shown
in Figure 3. From the results we can see that BYTE is more
robust to the detection score threshold than SORT. This is
because the second association in BYTE recovers the ob-
jects whose scores are lower than τhigh, and thus consid-
ers almost every detection box regardless of the change of
τhigh.
Analysis on low score detection boxes. To prove the ef-
fectiveness of BYTE, we collect the number of TPs and
FPs in the low score boxes obtained by BYTE. We use the
half training set of MOT17 and CrowdHuman for training
and evaluate on the half validation set of MOT17. First,
we keep all the low score detection boxes whose scores
range from τlow toτhigh and classify the TPs and FPs us-
ing ground truth annotations. Then, we select the tracking
results obtained by BYTE from low score detection boxes.
The results of each sequence are shown in Figure 4. We
can see that BYTE obtains notably more TPs than FPs from