[BugFix] Fix replay buffer extension with lists

[vmoens]

Fixes #1934

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/rl/1937

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

✅ You can merge normally! (3 Unrelated Failures)

As of commit 8961a04 with merge base 78b31a9 ():

FLAKY - The following jobs failed but were likely due to flakiness present on trunk:

• Habitat Tests on Linux / tests (3.9, 11.6) / linux-job (gh)
  RuntimeError: Command docker exec -t edadf8bb684f3e9dfe3e2f9233dd6d0fdb8e7d1b2f56b4e0fc6bbce1def4be2b /exec failed with exit code 139
• Unit-tests on Windows / unittests-cpu / windows-job (gh)
  The process 'C:\Program Files\Git\cmd\git.exe' failed with exit code 128

BROKEN TRUNK - The following job failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• Unit-tests on Windows / unittests-gpu / windows-job (gh)
  ##[error]The operation was canceled.

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[github-actions]

'&' can not be used here

$\color{#D29922}\textsf{\Large⚠\kern{0.2cm}\normalsize Warning}$

Result of CPU Benchmark Tests

Total Benchmarks: 89. Improved: $3$. Worsened: $3$.

Expand to view detailed results

Name	Max	Mean	Ops	Ops on Repo HEAD	Change
test_single	63.5724ms	62.8974ms	15.8989 Ops/s	15.0370 Ops/s	$\mathbf{\text{+5.73\%}}$
test_sync	55.8771ms	40.0102ms	24.9936 Ops/s	29.1461 Ops/s	$\mathbf{\text{-14.25\%}}$
test_async	58.1784ms	31.9971ms	31.2528 Ops/s	31.6580 Ops/s	$-1.28\mathrm{\%}$
test_simple	0.4997s	0.4396s	2.2749 Ops/s	2.2472 Ops/s	$+1.23\mathrm{\%}$
test_transformed	0.6511s	0.6024s	1.6600 Ops/s	1.6538 Ops/s	$+0.38\mathrm{\%}$
test_serial	1.4971s	1.4454s	0.6918 Ops/s	0.6840 Ops/s	$+1.16\mathrm{\%}$
test_parallel	1.4485s	1.3856s	0.7217 Ops/s	0.7202 Ops/s	$+0.20\mathrm{\%}$
test_step_mdp_speed[True-True-True-True-True]	0.2565ms	21.2249μs	47.1145 KOps/s	46.4005 KOps/s	$+1.54\mathrm{\%}$
test_step_mdp_speed[True-True-True-True-False]	66.0130μs	12.6018μs	79.3538 KOps/s	77.8683 KOps/s	$+1.91\mathrm{\%}$
test_step_mdp_speed[True-True-True-False-True]	38.0710μs	12.2847μs	81.4017 KOps/s	80.2635 KOps/s	$+1.42\mathrm{\%}$
test_step_mdp_speed[True-True-True-False-False]	48.2900μs	7.3235μs	136.5466 KOps/s	133.5377 KOps/s	$+2.25\mathrm{\%}$
test_step_mdp_speed[True-True-False-True-True]	64.5600μs	22.2152μs	45.0142 KOps/s	44.0351 KOps/s	$+2.22\mathrm{\%}$
test_step_mdp_speed[True-True-False-True-False]	58.1580μs	13.8647μs	72.1255 KOps/s	70.4808 KOps/s	$+2.33\mathrm{\%}$
test_step_mdp_speed[True-True-False-False-True]	48.3000μs	13.4125μs	74.5571 KOps/s	73.3879 KOps/s	$+1.59\mathrm{\%}$
test_step_mdp_speed[True-True-False-False-False]	48.3800μs	8.6526μs	115.5717 KOps/s	115.3159 KOps/s	$+0.22\mathrm{\%}$
test_step_mdp_speed[True-False-True-True-True]	58.2690μs	23.6294μs	42.3201 KOps/s	41.3742 KOps/s	$+2.29\mathrm{\%}$
test_step_mdp_speed[True-False-True-True-False]	50.2040μs	15.3809μs	65.0159 KOps/s	64.7629 KOps/s	$+0.39\mathrm{\%}$
test_step_mdp_speed[True-False-True-False-True]	39.0130μs	13.3456μs	74.9308 KOps/s	72.5382 KOps/s	$+3.30\mathrm{\%}$
test_step_mdp_speed[True-False-True-False-False]	41.0570μs	8.6566μs	115.5187 KOps/s	113.9895 KOps/s	$+1.34\mathrm{\%}$
test_step_mdp_speed[True-False-False-True-True]	98.9050μs	24.7450μs	40.4123 KOps/s	39.5099 KOps/s	$+2.28\mathrm{\%}$
test_step_mdp_speed[True-False-False-True-False]	47.4680μs	16.3840μs	61.0353 KOps/s	59.9181 KOps/s	$+1.86\mathrm{\%}$
test_step_mdp_speed[True-False-False-False-True]	58.7890μs	14.5447μs	68.7537 KOps/s	67.3023 KOps/s	$+2.16\mathrm{\%}$
test_step_mdp_speed[True-False-False-False-False]	41.8580μs	9.7978μs	102.0636 KOps/s	101.2292 KOps/s	$+0.82\mathrm{\%}$
test_step_mdp_speed[False-True-True-True-True]	56.8970μs	23.6479μs	42.2870 KOps/s	41.2617 KOps/s	$+2.48\mathrm{\%}$
test_step_mdp_speed[False-True-True-True-False]	44.7140μs	15.2317μs	65.6526 KOps/s	64.7299 KOps/s	$+1.43\mathrm{\%}$
test_step_mdp_speed[False-True-True-False-True]	54.6920μs	15.6456μs	63.9156 KOps/s	63.4810 KOps/s	$+0.68\mathrm{\%}$
test_step_mdp_speed[False-True-True-False-False]	41.4870μs	9.8544μs	101.4771 KOps/s	101.1429 KOps/s	$+0.33\mathrm{\%}$
test_step_mdp_speed[False-True-False-True-True]	43.3410μs	25.1420μs	39.7741 KOps/s	38.9297 KOps/s	$+2.17\mathrm{\%}$
test_step_mdp_speed[False-True-False-True-False]	62.4160μs	16.4090μs	60.9420 KOps/s	60.0085 KOps/s	$+1.56\mathrm{\%}$
test_step_mdp_speed[False-True-False-False-True]	51.6060μs	16.7828μs	59.5847 KOps/s	58.6659 KOps/s	$+1.57\mathrm{\%}$
test_step_mdp_speed[False-True-False-False-False]	36.1270μs	11.1030μs	90.0659 KOps/s	90.1709 KOps/s	$-0.12\mathrm{\%}$
test_step_mdp_speed[False-False-True-True-True]	86.9020μs	26.2094μs	38.1543 KOps/s	37.6626 KOps/s	$+1.31\mathrm{\%}$
test_step_mdp_speed[False-False-True-True-False]	50.6240μs	17.6688μs	56.5970 KOps/s	55.6566 KOps/s	$+1.69\mathrm{\%}$
test_step_mdp_speed[False-False-True-False-True]	56.3450μs	16.7848μs	59.5778 KOps/s	58.2694 KOps/s	$+2.25\mathrm{\%}$
test_step_mdp_speed[False-False-True-False-False]	33.7330μs	11.0030μs	90.8842 KOps/s	89.3991 KOps/s	$+1.66\mathrm{\%}$
test_step_mdp_speed[False-False-False-True-True]	73.1660μs	26.7213μs	37.4233 KOps/s	36.0893 KOps/s	$+3.70\mathrm{\%}$
test_step_mdp_speed[False-False-False-True-False]	47.8090μs	18.7756μs	53.2606 KOps/s	52.5067 KOps/s	$+1.44\mathrm{\%}$
test_step_mdp_speed[False-False-False-False-True]	58.1490μs	17.7473μs	56.3466 KOps/s	55.7267 KOps/s	$+1.11\mathrm{\%}$
test_step_mdp_speed[False-False-False-False-False]	54.3710μs	12.1127μs	82.5578 KOps/s	82.2745 KOps/s	$+0.34\mathrm{\%}$
test_values[generalized_advantage_estimate-True-True]	12.1466ms	9.3821ms	106.5854 Ops/s	104.8315 Ops/s	$+1.67\mathrm{\%}$
test_values[vec_generalized_advantage_estimate-True-True]	37.9750ms	33.5766ms	29.7827 Ops/s	29.7690 Ops/s	$+0.05\mathrm{\%}$
test_values[td0_return_estimate-False-False]	0.2411ms	0.1772ms	5.6448 KOps/s	5.5279 KOps/s	$+2.11\mathrm{\%}$
test_values[td1_return_estimate-False-False]	26.9922ms	23.8670ms	41.8988 Ops/s	42.1521 Ops/s	$-0.60\mathrm{\%}$
test_values[vec_td1_return_estimate-False-False]	34.8336ms	33.5792ms	29.7803 Ops/s	29.7240 Ops/s	$+0.19\mathrm{\%}$
test_values[td_lambda_return_estimate-True-False]	37.1192ms	34.0780ms	29.3445 Ops/s	29.2904 Ops/s	$+0.18\mathrm{\%}$
test_values[vec_td_lambda_return_estimate-True-False]	34.6703ms	33.4247ms	29.9180 Ops/s	29.6721 Ops/s	$+0.83\mathrm{\%}$
test_gae_speed[generalized_advantage_estimate-False-1-512]	16.9476ms	8.5416ms	117.0739 Ops/s	121.9214 Ops/s	$-3.98\mathrm{\%}$
test_gae_speed[vec_generalized_advantage_estimate-True-1-512]	2.2851ms	1.9948ms	501.3001 Ops/s	494.8969 Ops/s	$+1.29\mathrm{\%}$
test_gae_speed[vec_generalized_advantage_estimate-False-1-512]	0.9736ms	0.3588ms	2.7871 KOps/s	2.8475 KOps/s	$-2.12\mathrm{\%}$
test_gae_speed[vec_generalized_advantage_estimate-True-32-512]	40.1384ms	39.5476ms	25.2860 Ops/s	24.5580 Ops/s	$+2.96\mathrm{\%}$
test_gae_speed[vec_generalized_advantage_estimate-False-32-512]	3.8769ms	3.0734ms	325.3686 Ops/s	326.2771 Ops/s	$-0.28\mathrm{\%}$
test_dqn_speed	71.9037ms	1.4678ms	681.2733 Ops/s	719.1079 Ops/s	$\mathbf{\text{-5.26\%}}$
test_ddpg_speed	3.4593ms	2.7267ms	366.7413 Ops/s	362.2411 Ops/s	$+1.24\mathrm{\%}$
test_sac_speed	9.3868ms	8.7854ms	113.8249 Ops/s	111.4544 Ops/s	$+2.13\mathrm{\%}$
test_redq_speed	15.8186ms	13.9113ms	71.8841 Ops/s	71.5724 Ops/s	$+0.44\mathrm{\%}$
test_redq_deprec_speed	15.4293ms	14.5730ms	68.6202 Ops/s	68.7465 Ops/s	$-0.18\mathrm{\%}$
test_td3_speed	9.4708ms	8.9842ms	111.3063 Ops/s	110.0083 Ops/s	$+1.18\mathrm{\%}$
test_cql_speed	38.4790ms	37.6221ms	26.5802 Ops/s	25.9504 Ops/s	$+2.43\mathrm{\%}$
test_a2c_speed	9.0390ms	7.7666ms	128.7570 Ops/s	127.4734 Ops/s	$+1.01\mathrm{\%}$
test_ppo_speed	8.7937ms	8.1130ms	123.2592 Ops/s	121.8875 Ops/s	$+1.13\mathrm{\%}$
test_reinforce_speed	8.1912ms	6.8883ms	145.1739 Ops/s	143.6455 Ops/s	$+1.06\mathrm{\%}$
test_iql_speed	35.5342ms	33.8794ms	29.5165 Ops/s	29.2274 Ops/s	$+0.99\mathrm{\%}$
test_rb_sample[TensorDictReplayBuffer-ListStorage-RandomSampler-4000]	5.7312ms	3.0786ms	324.8187 Ops/s	320.9715 Ops/s	$+1.20\mathrm{\%}$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000]	0.6506ms	0.5279ms	1.8944 KOps/s	1.8040 KOps/s	$\mathbf{\text{+5.01\%}}$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000]	0.7387ms	0.4964ms	2.0144 KOps/s	2.0044 KOps/s	$+0.50\mathrm{\%}$
test_rb_sample[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000]	4.3520ms	2.9913ms	334.3005 Ops/s	327.6026 Ops/s	$+2.04\mathrm{\%}$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000]	0.8185ms	0.5273ms	1.8964 KOps/s	1.9071 KOps/s	$-0.56\mathrm{\%}$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000]	0.8594ms	0.5067ms	1.9735 KOps/s	2.0044 KOps/s	$-1.54\mathrm{\%}$
test_rb_sample[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000]	4.0723ms	3.0889ms	323.7420 Ops/s	313.0974 Ops/s	$+3.40\mathrm{\%}$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000]	0.7892ms	0.6457ms	1.5488 KOps/s	1.4955 KOps/s	$+3.56\mathrm{\%}$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000]	0.9457ms	0.6166ms	1.6218 KOps/s	1.6121 KOps/s	$+0.60\mathrm{\%}$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-RandomSampler-4000]	3.2407ms	2.9614ms	337.6834 Ops/s	319.6455 Ops/s	$\mathbf{\text{+5.64\%}}$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000]	0.9098ms	0.5751ms	1.7390 KOps/s	1.8935 KOps/s	$\mathbf{\text{-8.16\%}}$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000]	0.6048ms	0.5046ms	1.9819 KOps/s	1.9804 KOps/s	$+0.08\mathrm{\%}$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000]	3.2770ms	2.9854ms	334.9690 Ops/s	330.6664 Ops/s	$+1.30\mathrm{\%}$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000]	0.9159ms	0.5341ms	1.8722 KOps/s	1.9016 KOps/s	$-1.54\mathrm{\%}$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000]	0.6293ms	0.4974ms	2.0104 KOps/s	1.9785 KOps/s	$+1.61\mathrm{\%}$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000]	4.5126ms	3.1053ms	322.0347 Ops/s	312.5755 Ops/s	$+3.03\mathrm{\%}$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000]	0.9122ms	0.6474ms	1.5447 KOps/s	1.5410 KOps/s	$+0.24\mathrm{\%}$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000]	1.0026ms	0.6181ms	1.6177 KOps/s	1.6139 KOps/s	$+0.24\mathrm{\%}$
test_rb_populate[TensorDictReplayBuffer-ListStorage-RandomSampler-400]	0.1169s	8.3214ms	120.1723 Ops/s	119.4851 Ops/s	$+0.58\mathrm{\%}$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-400]	16.0769ms	13.4690ms	74.2444 Ops/s	75.1520 Ops/s	$-1.21\mathrm{\%}$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-400]	5.3255ms	2.6033ms	384.1211 Ops/s	389.9008 Ops/s	$-1.48\mathrm{\%}$
test_rb_populate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-400]	0.1031s	9.9250ms	100.7552 Ops/s	99.3968 Ops/s	$+1.37\mathrm{\%}$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-400]	17.5899ms	13.5082ms	74.0291 Ops/s	74.2237 Ops/s	$-0.26\mathrm{\%}$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-400]	4.9626ms	2.5601ms	390.6094 Ops/s	385.6214 Ops/s	$+1.29\mathrm{\%}$
test_rb_populate[TensorDictPrioritizedReplayBuffer-ListStorage-None-400]	0.1025s	10.2461ms	97.5983 Ops/s	95.2585 Ops/s	$+2.46\mathrm{\%}$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-400]	16.6291ms	13.7402ms	72.7794 Ops/s	72.5331 Ops/s	$+0.34\mathrm{\%}$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-400]	8.2610ms	2.9368ms	340.5107 Ops/s	347.6636 Ops/s	$-2.06\mathrm{\%}$

[github-actions]

'&' can not be used here

$\color{#D29922}\textsf{\Large⚠\kern{0.2cm}\normalsize Warning}$

Result of GPU Benchmark Tests

Total Benchmarks: 92. Improved: $1$. Worsened: $3$.

Expand to view detailed results

Name	Max	Mean	Ops	Ops on Repo HEAD	Change
test_single	0.1131s	0.1128s	8.8684 Ops/s	8.4686 Ops/s	$+4.72\mathrm{\%}$
test_sync	95.3914ms	95.0650ms	10.5191 Ops/s	10.4611 Ops/s	$+0.55\mathrm{\%}$
test_async	0.1814s	91.8895ms	10.8826 Ops/s	10.9695 Ops/s	$-0.79\mathrm{\%}$
test_single_pixels	0.2030s	0.1418s	7.0509 Ops/s	8.0400 Ops/s	$\mathbf{\text{-12.30\%}}$
test_sync_pixels	83.8452ms	82.5985ms	12.1068 Ops/s	12.5569 Ops/s	$-3.59\mathrm{\%}$
test_async_pixels	0.1441s	73.2673ms	13.6487 Ops/s	13.2039 Ops/s	$+3.37\mathrm{\%}$
test_simple	0.8089s	0.8080s	1.2377 Ops/s	1.2243 Ops/s	$+1.09\mathrm{\%}$
test_transformed	1.0382s	1.0377s	0.9637 Ops/s	0.9579 Ops/s	$+0.60\mathrm{\%}$
test_serial	2.5059s	2.4482s	0.4085 Ops/s	0.4111 Ops/s	$-0.64\mathrm{\%}$
test_parallel	2.1396s	2.1030s	0.4755 Ops/s	0.4868 Ops/s	$-2.32\mathrm{\%}$
test_step_mdp_speed[True-True-True-True-True]	0.1120ms	32.7749μs	30.5112 KOps/s	30.1299 KOps/s	$+1.27\mathrm{\%}$
test_step_mdp_speed[True-True-True-True-False]	0.1410ms	19.7460μs	50.6432 KOps/s	51.0431 KOps/s	$-0.78\mathrm{\%}$
test_step_mdp_speed[True-True-True-False-True]	0.1657ms	18.7030μs	53.4672 KOps/s	53.6744 KOps/s	$-0.39\mathrm{\%}$
test_step_mdp_speed[True-True-True-False-False]	26.8010μs	11.1271μs	89.8704 KOps/s	89.4999 KOps/s	$+0.41\mathrm{\%}$
test_step_mdp_speed[True-True-False-True-True]	81.5240μs	34.9199μs	28.6369 KOps/s	28.9378 KOps/s	$-1.04\mathrm{\%}$
test_step_mdp_speed[True-True-False-True-False]	39.9420μs	21.5971μs	46.3024 KOps/s	46.3031 KOps/s	$-0.00\mathrm{\%}$
test_step_mdp_speed[True-True-False-False-True]	61.7430μs	20.5487μs	48.6649 KOps/s	48.7983 KOps/s	$-0.27\mathrm{\%}$
test_step_mdp_speed[True-True-False-False-False]	38.3520μs	13.0066μs	76.8839 KOps/s	75.6865 KOps/s	$+1.58\mathrm{\%}$
test_step_mdp_speed[True-False-True-True-True]	54.8520μs	36.9516μs	27.0624 KOps/s	26.9287 KOps/s	$+0.50\mathrm{\%}$
test_step_mdp_speed[True-False-True-True-False]	43.2020μs	23.3387μs	42.8474 KOps/s	42.5833 KOps/s	$+0.62\mathrm{\%}$
test_step_mdp_speed[True-False-True-False-True]	44.5630μs	20.3813μs	49.0645 KOps/s	49.0188 KOps/s	$+0.09\mathrm{\%}$
test_step_mdp_speed[True-False-True-False-False]	33.9210μs	13.0922μs	76.3816 KOps/s	75.2847 KOps/s	$+1.46\mathrm{\%}$
test_step_mdp_speed[True-False-False-True-True]	59.8230μs	38.4440μs	26.0119 KOps/s	25.5896 KOps/s	$+1.65\mathrm{\%}$
test_step_mdp_speed[True-False-False-True-False]	0.2170ms	25.3305μs	39.4781 KOps/s	39.3653 KOps/s	$+0.29\mathrm{\%}$
test_step_mdp_speed[True-False-False-False-True]	39.7110μs	22.4071μs	44.6288 KOps/s	45.4961 KOps/s	$-1.91\mathrm{\%}$
test_step_mdp_speed[True-False-False-False-False]	29.9620μs	14.7846μs	67.6381 KOps/s	66.6219 KOps/s	$+1.53\mathrm{\%}$
test_step_mdp_speed[False-True-True-True-True]	62.2130μs	36.4335μs	27.4473 KOps/s	27.1824 KOps/s	$+0.97\mathrm{\%}$
test_step_mdp_speed[False-True-True-True-False]	41.6020μs	23.6588μs	42.2675 KOps/s	42.6956 KOps/s	$-1.00\mathrm{\%}$
test_step_mdp_speed[False-True-True-False-True]	50.4230μs	24.2357μs	41.2614 KOps/s	40.6482 KOps/s	$+1.51\mathrm{\%}$
test_step_mdp_speed[False-True-True-False-False]	30.7710μs	14.8283μs	67.4387 KOps/s	66.7328 KOps/s	$+1.06\mathrm{\%}$
test_step_mdp_speed[False-True-False-True-True]	81.6840μs	38.4560μs	26.0038 KOps/s	25.6354 KOps/s	$+1.44\mathrm{\%}$
test_step_mdp_speed[False-True-False-True-False]	44.9930μs	25.4686μs	39.2640 KOps/s	39.0527 KOps/s	$+0.54\mathrm{\%}$
test_step_mdp_speed[False-True-False-False-True]	41.9420μs	26.3383μs	37.9675 KOps/s	37.9841 KOps/s	$-0.04\mathrm{\%}$
test_step_mdp_speed[False-True-False-False-False]	31.0810μs	16.7438μs	59.7235 KOps/s	59.8567 KOps/s	$-0.22\mathrm{\%}$
test_step_mdp_speed[False-False-True-True-True]	72.1540μs	40.2200μs	24.8632 KOps/s	24.7980 KOps/s	$+0.26\mathrm{\%}$
test_step_mdp_speed[False-False-True-True-False]	45.5620μs	27.4922μs	36.3740 KOps/s	36.5150 KOps/s	$-0.39\mathrm{\%}$
test_step_mdp_speed[False-False-True-False-True]	42.4420μs	26.2391μs	38.1110 KOps/s	38.1874 KOps/s	$-0.20\mathrm{\%}$
test_step_mdp_speed[False-False-True-False-False]	37.6310μs	16.7458μs	59.7164 KOps/s	59.2016 KOps/s	$+0.87\mathrm{\%}$
test_step_mdp_speed[False-False-False-True-True]	68.6940μs	41.8831μs	23.8760 KOps/s	23.7422 KOps/s	$+0.56\mathrm{\%}$
test_step_mdp_speed[False-False-False-True-False]	52.7930μs	28.8173μs	34.7014 KOps/s	34.4525 KOps/s	$+0.72\mathrm{\%}$
test_step_mdp_speed[False-False-False-False-True]	59.9630μs	27.6396μs	36.1800 KOps/s	35.7711 KOps/s	$+1.14\mathrm{\%}$
test_step_mdp_speed[False-False-False-False-False]	32.4320μs	18.4231μs	54.2798 KOps/s	54.1315 KOps/s	$+0.27\mathrm{\%}$
test_values[generalized_advantage_estimate-True-True]	24.6753ms	24.2394ms	41.2552 Ops/s	39.9394 Ops/s	$+3.29\mathrm{\%}$
test_values[vec_generalized_advantage_estimate-True-True]	83.2971ms	3.2232ms	310.2483 Ops/s	310.7337 Ops/s	$-0.16\mathrm{\%}$
test_values[td0_return_estimate-False-False]	99.0250μs	59.4644μs	16.8168 KOps/s	17.2989 KOps/s	$-2.79\mathrm{\%}$
test_values[td1_return_estimate-False-False]	51.9144ms	51.5386ms	19.4029 Ops/s	19.6993 Ops/s	$-1.50\mathrm{\%}$
test_values[vec_td1_return_estimate-False-False]	2.0834ms	1.7515ms	570.9372 Ops/s	573.0319 Ops/s	$-0.37\mathrm{\%}$
test_values[td_lambda_return_estimate-True-False]	83.1412ms	82.7015ms	12.0917 Ops/s	12.2859 Ops/s	$-1.58\mathrm{\%}$
test_values[vec_td_lambda_return_estimate-True-False]	3.9259ms	1.7846ms	560.3606 Ops/s	561.3487 Ops/s	$-0.18\mathrm{\%}$
test_gae_speed[generalized_advantage_estimate-False-1-512]	23.1110ms	22.8338ms	43.7947 Ops/s	44.6173 Ops/s	$-1.84\mathrm{\%}$
test_gae_speed[vec_generalized_advantage_estimate-True-1-512]	0.8693ms	0.6903ms	1.4487 KOps/s	1.4432 KOps/s	$+0.38\mathrm{\%}$
test_gae_speed[vec_generalized_advantage_estimate-False-1-512]	0.6852ms	0.6399ms	1.5628 KOps/s	1.5369 KOps/s	$+1.69\mathrm{\%}$
test_gae_speed[vec_generalized_advantage_estimate-True-32-512]	1.6018ms	1.4446ms	692.2235 Ops/s	696.4400 Ops/s	$-0.61\mathrm{\%}$
test_gae_speed[vec_generalized_advantage_estimate-False-32-512]	0.9424ms	0.6615ms	1.5117 KOps/s	1.5343 KOps/s	$-1.47\mathrm{\%}$
test_dqn_speed	3.9452ms	1.4445ms	692.2688 Ops/s	645.8949 Ops/s	$\mathbf{\text{+7.18\%}}$
test_ddpg_speed	2.9466ms	2.7345ms	365.6909 Ops/s	361.1414 Ops/s	$+1.26\mathrm{\%}$
test_sac_speed	9.0094ms	8.5061ms	117.5621 Ops/s	116.7804 Ops/s	$+0.67\mathrm{\%}$
test_redq_speed	12.5932ms	10.6356ms	94.0236 Ops/s	93.7034 Ops/s	$+0.34\mathrm{\%}$
test_redq_deprec_speed	12.5078ms	11.7827ms	84.8698 Ops/s	82.4138 Ops/s	$+2.98\mathrm{\%}$
test_td3_speed	10.4524ms	8.6837ms	115.1584 Ops/s	115.0680 Ops/s	$+0.08\mathrm{\%}$
test_cql_speed	28.1114ms	25.8344ms	38.7081 Ops/s	37.9941 Ops/s	$+1.88\mathrm{\%}$
test_a2c_speed	7.2356ms	5.5297ms	180.8431 Ops/s	176.1199 Ops/s	$+2.68\mathrm{\%}$
test_ppo_speed	6.0653ms	5.8237ms	171.7112 Ops/s	165.5896 Ops/s	$+3.70\mathrm{\%}$
test_reinforce_speed	5.2643ms	4.5025ms	222.1013 Ops/s	213.9263 Ops/s	$+3.82\mathrm{\%}$
test_iql_speed	20.4890ms	19.6539ms	50.8805 Ops/s	49.2180 Ops/s	$+3.38\mathrm{\%}$
test_rb_sample[TensorDictReplayBuffer-ListStorage-RandomSampler-4000]	3.8064ms	3.5540ms	281.3731 Ops/s	280.5757 Ops/s	$+0.28\mathrm{\%}$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000]	0.7316ms	0.5585ms	1.7904 KOps/s	1.7799 KOps/s	$+0.59\mathrm{\%}$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000]	0.7303ms	0.5320ms	1.8798 KOps/s	1.8793 KOps/s	$+0.03\mathrm{\%}$
test_rb_sample[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000]	3.8358ms	3.5771ms	279.5528 Ops/s	278.6175 Ops/s	$+0.34\mathrm{\%}$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000]	0.7197ms	0.5507ms	1.8157 KOps/s	1.7965 KOps/s	$+1.07\mathrm{\%}$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000]	0.7156ms	0.5251ms	1.9044 KOps/s	1.8804 KOps/s	$+1.28\mathrm{\%}$
test_rb_sample[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000]	3.8568ms	3.6813ms	271.6432 Ops/s	269.5669 Ops/s	$+0.77\mathrm{\%}$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000]	91.7310ms	0.7653ms	1.3066 KOps/s	1.4497 KOps/s	$\mathbf{\text{-9.87\%}}$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000]	0.8462ms	0.6579ms	1.5200 KOps/s	1.5132 KOps/s	$+0.45\mathrm{\%}$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-RandomSampler-4000]	3.7088ms	3.5359ms	282.8106 Ops/s	282.5487 Ops/s	$+0.09\mathrm{\%}$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000]	0.8019ms	0.5599ms	1.7859 KOps/s	1.7582 KOps/s	$+1.58\mathrm{\%}$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000]	0.6991ms	0.5330ms	1.8761 KOps/s	1.8635 KOps/s	$+0.67\mathrm{\%}$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000]	3.8639ms	3.5773ms	279.5439 Ops/s	280.6742 Ops/s	$-0.40\mathrm{\%}$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000]	0.7512ms	0.5540ms	1.8051 KOps/s	1.7910 KOps/s	$+0.79\mathrm{\%}$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000]	0.7604ms	0.5272ms	1.8969 KOps/s	1.8727 KOps/s	$+1.29\mathrm{\%}$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000]	3.8582ms	3.6888ms	271.0930 Ops/s	271.5179 Ops/s	$-0.16\mathrm{\%}$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000]	0.8684ms	0.6854ms	1.4590 KOps/s	1.4477 KOps/s	$+0.78\mathrm{\%}$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000]	0.8605ms	0.6604ms	1.5142 KOps/s	1.5001 KOps/s	$+0.94\mathrm{\%}$
test_rb_populate[TensorDictReplayBuffer-ListStorage-RandomSampler-400]	0.1003s	11.0041ms	90.8752 Ops/s	102.4908 Ops/s	$\mathbf{\text{-11.33\%}}$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-400]	22.2253ms	15.8904ms	62.9309 Ops/s	62.6632 Ops/s	$+0.43\mathrm{\%}$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-400]	5.4731ms	3.0233ms	330.7617 Ops/s	325.3495 Ops/s	$+1.66\mathrm{\%}$
test_rb_populate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-400]	99.6734ms	10.9914ms	90.9799 Ops/s	88.8815 Ops/s	$+2.36\mathrm{\%}$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-400]	17.9838ms	15.7290ms	63.5767 Ops/s	62.3388 Ops/s	$+1.99\mathrm{\%}$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-400]	5.8129ms	3.0270ms	330.3582 Ops/s	326.7188 Ops/s	$+1.11\mathrm{\%}$
test_rb_populate[TensorDictPrioritizedReplayBuffer-ListStorage-None-400]	99.5977ms	11.3283ms	88.2743 Ops/s	86.1918 Ops/s	$+2.42\mathrm{\%}$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-400]	18.3134ms	15.9962ms	62.5147 Ops/s	61.4014 Ops/s	$+1.81\mathrm{\%}$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-400]	6.4467ms	3.3089ms	302.2109 Ops/s	296.6758 Ops/s	$+1.87\mathrm{\%}$