[Minor] Use the main branch for the M1 build wheels

[DanilBaibak]

Description

Use the main branch for the M1 build wheels workflow.

Motivation and Context

All workflows should use the main branch, except for debugging purposes.

• I have raised an issue to propose this change (required for new features and bug fixes)

Types of changes

What types of changes does your code introduce? Remove all that do not apply:

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds core functionality)
• Breaking change (fix or feature that would cause existing functionality to change)
• Documentation (update in the documentation)
• Example (update in the folder of examples)

Checklist

Go over all the following points, and put an x in all the boxes that apply.
If you are unsure about any of these, don't hesitate to ask. We are here to help!

• I have read the CONTRIBUTION guide (required)
• My change requires a change to the documentation.
• I have updated the tests accordingly (required for a bug fix or a new feature).
• I have updated the documentation accordingly.

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/rl/1965

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

✅ You can merge normally! (1 Unrelated Failure)

As of commit 54b455c with merge base 8f04818 ():

BROKEN TRUNK - The following job failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• Unit-tests on Windows / unittests-gpu / windows-job (gh)
  ##[error]The operation was canceled.

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[github-actions]

'&' can not be used here

$\color{#D29922}\textsf{\Large⚠\kern{0.2cm}\normalsize Warning}$

Result of CPU Benchmark Tests

Total Benchmarks: 89. Improved: $3$. Worsened: $4$.

Expand to view detailed results

Name	Max	Mean	Ops	Ops on Repo HEAD	Change
test_single	63.9696ms	61.5685ms	16.2421 Ops/s	16.1802 Ops/s	$+0.38\mathrm{\%}$
test_sync	34.8820ms	33.0214ms	30.2834 Ops/s	29.3798 Ops/s	$+3.08\mathrm{\%}$
test_async	67.8678ms	31.1989ms	32.0524 Ops/s	32.5186 Ops/s	$-1.43\mathrm{\%}$
test_simple	0.4942s	0.4380s	2.2830 Ops/s	2.3306 Ops/s	$-2.04\mathrm{\%}$
test_transformed	0.6430s	0.5903s	1.6941 Ops/s	1.7137 Ops/s	$-1.14\mathrm{\%}$
test_serial	1.5146s	1.4492s	0.6901 Ops/s	0.6933 Ops/s	$-0.46\mathrm{\%}$
test_parallel	1.5031s	1.4374s	0.6957 Ops/s	0.6999 Ops/s	$-0.61\mathrm{\%}$
test_step_mdp_speed[True-True-True-True-True]	98.4340μs	21.6088μs	46.2775 KOps/s	46.0899 KOps/s	$+0.41\mathrm{\%}$
test_step_mdp_speed[True-True-True-True-False]	38.1710μs	13.1605μs	75.9847 KOps/s	74.6308 KOps/s	$+1.81\mathrm{\%}$
test_step_mdp_speed[True-True-True-False-True]	41.0370μs	12.7691μs	78.3138 KOps/s	79.1468 KOps/s	$-1.05\mathrm{\%}$
test_step_mdp_speed[True-True-True-False-False]	46.7650μs	7.6020μs	131.5442 KOps/s	129.0680 KOps/s	$+1.92\mathrm{\%}$
test_step_mdp_speed[True-True-False-True-True]	49.0820μs	22.7411μs	43.9733 KOps/s	42.9808 KOps/s	$+2.31\mathrm{\%}$
test_step_mdp_speed[True-True-False-True-False]	85.2400μs	14.2846μs	70.0054 KOps/s	68.4638 KOps/s	$+2.25\mathrm{\%}$
test_step_mdp_speed[True-True-False-False-True]	39.1440μs	13.8973μs	71.9564 KOps/s	71.2161 KOps/s	$+1.04\mathrm{\%}$
test_step_mdp_speed[True-True-False-False-False]	56.9370μs	9.0408μs	110.6096 KOps/s	110.0484 KOps/s	$+0.51\mathrm{\%}$
test_step_mdp_speed[True-False-True-True-True]	51.0450μs	24.4407μs	40.9154 KOps/s	40.1444 KOps/s	$+1.92\mathrm{\%}$
test_step_mdp_speed[True-False-True-True-False]	54.8330μs	15.9287μs	62.7796 KOps/s	62.4533 KOps/s	$+0.52\mathrm{\%}$
test_step_mdp_speed[True-False-True-False-True]	57.9580μs	13.9568μs	71.6495 KOps/s	70.8537 KOps/s	$+1.12\mathrm{\%}$
test_step_mdp_speed[True-False-True-False-False]	31.6590μs	8.9389μs	111.8707 KOps/s	110.2891 KOps/s	$+1.43\mathrm{\%}$
test_step_mdp_speed[True-False-False-True-True]	52.3380μs	25.3392μs	39.4646 KOps/s	38.8491 KOps/s	$+1.58\mathrm{\%}$
test_step_mdp_speed[True-False-False-True-False]	40.2850μs	17.1606μs	58.2730 KOps/s	57.5251 KOps/s	$+1.30\mathrm{\%}$
test_step_mdp_speed[True-False-False-False-True]	50.4650μs	15.1696μs	65.9214 KOps/s	65.5012 KOps/s	$+0.64\mathrm{\%}$
test_step_mdp_speed[True-False-False-False-False]	28.1030μs	10.2668μs	97.4011 KOps/s	95.9559 KOps/s	$+1.51\mathrm{\%}$
test_step_mdp_speed[False-True-True-True-True]	72.2040μs	24.2684μs	41.2058 KOps/s	40.3103 KOps/s	$+2.22\mathrm{\%}$
test_step_mdp_speed[False-True-True-True-False]	38.0710μs	15.8671μs	63.0235 KOps/s	61.7992 KOps/s	$+1.98\mathrm{\%}$
test_step_mdp_speed[False-True-True-False-True]	59.5220μs	16.0952μs	62.1302 KOps/s	60.1260 KOps/s	$+3.33\mathrm{\%}$
test_step_mdp_speed[False-True-True-False-False]	25.3980μs	10.2590μs	97.4754 KOps/s	95.4640 KOps/s	$+2.11\mathrm{\%}$
test_step_mdp_speed[False-True-False-True-True]	38.8930μs	26.2241μs	38.1328 KOps/s	37.8629 KOps/s	$+0.71\mathrm{\%}$
test_step_mdp_speed[False-True-False-True-False]	55.0530μs	17.0898μs	58.5144 KOps/s	57.5703 KOps/s	$+1.64\mathrm{\%}$
test_step_mdp_speed[False-True-False-False-True]	62.5370μs	17.3668μs	57.5812 KOps/s	56.3884 KOps/s	$+2.12\mathrm{\%}$
test_step_mdp_speed[False-True-False-False-False]	38.9830μs	11.4000μs	87.7194 KOps/s	85.8322 KOps/s	$+2.20\mathrm{\%}$
test_step_mdp_speed[False-False-True-True-True]	77.8550μs	26.8214μs	37.2836 KOps/s	36.4882 KOps/s	$+2.18\mathrm{\%}$
test_step_mdp_speed[False-False-True-True-False]	47.3980μs	18.4803μs	54.1117 KOps/s	53.1197 KOps/s	$+1.87\mathrm{\%}$
test_step_mdp_speed[False-False-True-False-True]	64.4600μs	17.4702μs	57.2405 KOps/s	56.1102 KOps/s	$+2.01\mathrm{\%}$
test_step_mdp_speed[False-False-True-False-False]	29.1050μs	11.5012μs	86.9475 KOps/s	85.0752 KOps/s	$+2.20\mathrm{\%}$
test_step_mdp_speed[False-False-False-True-True]	62.4470μs	27.6996μs	36.1016 KOps/s	34.9750 KOps/s	$+3.22\mathrm{\%}$
test_step_mdp_speed[False-False-False-True-False]	67.7960μs	19.8932μs	50.2685 KOps/s	50.3559 KOps/s	$-0.17\mathrm{\%}$
test_step_mdp_speed[False-False-False-False-True]	37.7000μs	18.4009μs	54.3450 KOps/s	53.1203 KOps/s	$+2.31\mathrm{\%}$
test_step_mdp_speed[False-False-False-False-False]	52.1780μs	12.6746μs	78.8978 KOps/s	78.8142 KOps/s	$+0.11\mathrm{\%}$
test_values[generalized_advantage_estimate-True-True]	9.6083ms	9.3140ms	107.3654 Ops/s	108.8125 Ops/s	$-1.33\mathrm{\%}$
test_values[vec_generalized_advantage_estimate-True-True]	48.9164ms	34.2436ms	29.2025 Ops/s	29.6848 Ops/s	$-1.62\mathrm{\%}$
test_values[td0_return_estimate-False-False]	0.2403ms	0.1651ms	6.0578 KOps/s	6.1141 KOps/s	$-0.92\mathrm{\%}$
test_values[td1_return_estimate-False-False]	23.7217ms	23.2176ms	43.0708 Ops/s	44.1369 Ops/s	$-2.42\mathrm{\%}$
test_values[vec_td1_return_estimate-False-False]	35.4254ms	33.6414ms	29.7253 Ops/s	29.4973 Ops/s	$+0.77\mathrm{\%}$
test_values[td_lambda_return_estimate-True-False]	36.7512ms	33.5975ms	29.7641 Ops/s	30.2187 Ops/s	$-1.50\mathrm{\%}$
test_values[vec_td_lambda_return_estimate-True-False]	34.8948ms	33.5886ms	29.7720 Ops/s	29.6043 Ops/s	$+0.57\mathrm{\%}$
test_gae_speed[generalized_advantage_estimate-False-1-512]	8.8733ms	8.2051ms	121.8754 Ops/s	122.8859 Ops/s	$-0.82\mathrm{\%}$
test_gae_speed[vec_generalized_advantage_estimate-True-1-512]	1.9249ms	1.7961ms	556.7763 Ops/s	563.6292 Ops/s	$-1.22\mathrm{\%}$
test_gae_speed[vec_generalized_advantage_estimate-False-1-512]	0.6520ms	0.3607ms	2.7725 KOps/s	2.8236 KOps/s	$-1.81\mathrm{\%}$
test_gae_speed[vec_generalized_advantage_estimate-True-32-512]	47.5110ms	44.7269ms	22.3579 Ops/s	21.4994 Ops/s	$+3.99\mathrm{\%}$
test_gae_speed[vec_generalized_advantage_estimate-False-32-512]	3.7767ms	3.0266ms	330.4073 Ops/s	331.8785 Ops/s	$-0.44\mathrm{\%}$
test_dqn_speed	70.5545ms	1.5451ms	647.2271 Ops/s	711.2547 Ops/s	$\mathbf{\text{-9.00\%}}$
test_ddpg_speed	3.5209ms	2.8241ms	354.0947 Ops/s	357.3779 Ops/s	$-0.92\mathrm{\%}$
test_sac_speed	9.7666ms	8.4600ms	118.2031 Ops/s	119.9205 Ops/s	$-1.43\mathrm{\%}$
test_redq_speed	15.5995ms	13.7066ms	72.9576 Ops/s	75.0486 Ops/s	$-2.79\mathrm{\%}$
test_redq_deprec_speed	14.2975ms	13.3423ms	74.9496 Ops/s	74.0412 Ops/s	$+1.23\mathrm{\%}$
test_td3_speed	8.8406ms	8.4713ms	118.0457 Ops/s	119.7257 Ops/s	$-1.40\mathrm{\%}$
test_cql_speed	38.2792ms	36.5647ms	27.3488 Ops/s	27.4663 Ops/s	$-0.43\mathrm{\%}$
test_a2c_speed	8.0443ms	7.3889ms	135.3381 Ops/s	135.9744 Ops/s	$-0.47\mathrm{\%}$
test_ppo_speed	8.7409ms	7.6912ms	130.0186 Ops/s	130.3072 Ops/s	$-0.22\mathrm{\%}$
test_reinforce_speed	7.6925ms	6.6244ms	150.9574 Ops/s	151.5113 Ops/s	$-0.37\mathrm{\%}$
test_iql_speed	34.4869ms	32.9753ms	30.3257 Ops/s	30.3280 Ops/s	$-0.01\mathrm{\%}$
test_rb_sample[TensorDictReplayBuffer-ListStorage-RandomSampler-4000]	2.4506ms	2.2143ms	451.6100 Ops/s	443.7426 Ops/s	$+1.77\mathrm{\%}$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000]	1.0868ms	0.4959ms	2.0166 KOps/s	2.0308 KOps/s	$-0.70\mathrm{\%}$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000]	3.5345ms	0.4800ms	2.0833 KOps/s	2.1418 KOps/s	$-2.73\mathrm{\%}$
test_rb_sample[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000]	2.4849ms	2.1997ms	454.6010 Ops/s	454.8449 Ops/s	$-0.05\mathrm{\%}$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000]	0.8931ms	0.4869ms	2.0538 KOps/s	2.0739 KOps/s	$-0.97\mathrm{\%}$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000]	0.7365ms	0.4647ms	2.1520 KOps/s	2.1596 KOps/s	$-0.35\mathrm{\%}$
test_rb_sample[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000]	3.5708ms	2.3014ms	434.5161 Ops/s	426.2065 Ops/s	$+1.95\mathrm{\%}$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000]	0.9803ms	0.6068ms	1.6479 KOps/s	1.6443 KOps/s	$+0.22\mathrm{\%}$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000]	0.8700ms	0.5815ms	1.7198 KOps/s	1.7182 KOps/s	$+0.09\mathrm{\%}$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-RandomSampler-4000]	3.3198ms	2.1795ms	458.8310 Ops/s	467.4952 Ops/s	$-1.85\mathrm{\%}$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000]	0.9364ms	0.4984ms	2.0065 KOps/s	1.6516 KOps/s	$\mathbf{\text{+21.49\%}}$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000]	0.1046s	0.5781ms	1.7298 KOps/s	2.1283 KOps/s	$\mathbf{\text{-18.72\%}}$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000]	2.8101ms	2.2401ms	446.4182 Ops/s	456.7259 Ops/s	$-2.26\mathrm{\%}$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000]	0.6752ms	0.4896ms	2.0425 KOps/s	2.0445 KOps/s	$-0.09\mathrm{\%}$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000]	3.6179ms	0.4711ms	2.1229 KOps/s	2.1667 KOps/s	$-2.02\mathrm{\%}$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000]	2.6754ms	2.3215ms	430.7523 Ops/s	436.3679 Ops/s	$-1.29\mathrm{\%}$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000]	95.2464ms	0.7544ms	1.3256 KOps/s	1.6467 KOps/s	$\mathbf{\text{-19.50\%}}$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000]	0.8522ms	0.5820ms	1.7181 KOps/s	1.7219 KOps/s	$-0.22\mathrm{\%}$
test_rb_populate[TensorDictReplayBuffer-ListStorage-RandomSampler-400]	0.1008s	5.5213ms	181.1167 Ops/s	171.8535 Ops/s	$\mathbf{\text{+5.39\%}}$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-400]	15.3507ms	12.2504ms	81.6301 Ops/s	83.6368 Ops/s	$-2.40\mathrm{\%}$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-400]	1.5940ms	1.0810ms	925.0739 Ops/s	965.3056 Ops/s	$-4.17\mathrm{\%}$
test_rb_populate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-400]	95.6309ms	7.2483ms	137.9640 Ops/s	137.5540 Ops/s	$+0.30\mathrm{\%}$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-400]	15.7076ms	12.1683ms	82.1807 Ops/s	83.5154 Ops/s	$-1.60\mathrm{\%}$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-400]	1.6427ms	1.0915ms	916.1700 Ops/s	954.7365 Ops/s	$-4.04\mathrm{\%}$
test_rb_populate[TensorDictPrioritizedReplayBuffer-ListStorage-None-400]	95.8192ms	7.4927ms	133.4635 Ops/s	171.1057 Ops/s	$\mathbf{\text{-22.00\%}}$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-400]	14.8151ms	12.4223ms	80.5006 Ops/s	70.9334 Ops/s	$\mathbf{\text{+13.49\%}}$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-400]	4.2417ms	1.4725ms	679.1000 Ops/s	693.7131 Ops/s	$-2.11\mathrm{\%}$

[github-actions]

'&' can not be used here

$\color{#D29922}\textsf{\Large⚠\kern{0.2cm}\normalsize Warning}$

Result of GPU Benchmark Tests

Total Benchmarks: 92. Improved: $4$. Worsened: $5$.

Expand to view detailed results

Name	Max	Mean	Ops	Ops on Repo HEAD	Change
test_single	0.1166s	0.1164s	8.5946 Ops/s	8.6600 Ops/s	$-0.76\mathrm{\%}$
test_sync	96.3724ms	95.9278ms	10.4245 Ops/s	10.4168 Ops/s	$+0.07\mathrm{\%}$
test_async	0.1813s	91.7103ms	10.9039 Ops/s	10.9501 Ops/s	$-0.42\mathrm{\%}$
test_single_pixels	0.2115s	0.1480s	6.7587 Ops/s	7.3068 Ops/s	$\mathbf{\text{-7.50\%}}$
test_sync_pixels	85.8475ms	80.1326ms	12.4793 Ops/s	12.6518 Ops/s	$-1.36\mathrm{\%}$
test_async_pixels	0.1481s	67.5694ms	14.7996 Ops/s	15.8923 Ops/s	$\mathbf{\text{-6.88\%}}$
test_simple	0.8715s	0.8689s	1.1509 Ops/s	1.1944 Ops/s	$-3.64\mathrm{\%}$
test_transformed	1.1022s	1.0954s	0.9129 Ops/s	0.9365 Ops/s	$-2.52\mathrm{\%}$
test_serial	2.6860s	2.6397s	0.3788 Ops/s	0.4008 Ops/s	$\mathbf{\text{-5.48\%}}$
test_parallel	2.1675s	2.1352s	0.4683 Ops/s	0.4737 Ops/s	$-1.12\mathrm{\%}$
test_step_mdp_speed[True-True-True-True-True]	94.2710μs	32.8482μs	30.4430 KOps/s	29.8128 KOps/s	$+2.11\mathrm{\%}$
test_step_mdp_speed[True-True-True-True-False]	42.0710μs	19.8039μs	50.4951 KOps/s	49.0316 KOps/s	$+2.98\mathrm{\%}$
test_step_mdp_speed[True-True-True-False-True]	35.8910μs	18.5215μs	53.9914 KOps/s	52.8880 KOps/s	$+2.09\mathrm{\%}$
test_step_mdp_speed[True-True-True-False-False]	38.2610μs	11.2035μs	89.2581 KOps/s	87.5483 KOps/s	$+1.95\mathrm{\%}$
test_step_mdp_speed[True-True-False-True-True]	64.0810μs	34.7033μs	28.8157 KOps/s	28.3685 KOps/s	$+1.58\mathrm{\%}$
test_step_mdp_speed[True-True-False-True-False]	41.4800μs	21.7721μs	45.9304 KOps/s	44.6819 KOps/s	$+2.79\mathrm{\%}$
test_step_mdp_speed[True-True-False-False-True]	38.7200μs	20.4471μs	48.9066 KOps/s	48.1726 KOps/s	$+1.52\mathrm{\%}$
test_step_mdp_speed[True-True-False-False-False]	35.9910μs	13.1215μs	76.2106 KOps/s	74.9149 KOps/s	$+1.73\mathrm{\%}$
test_step_mdp_speed[True-False-True-True-True]	60.3010μs	37.2054μs	26.8778 KOps/s	27.1949 KOps/s	$-1.17\mathrm{\%}$
test_step_mdp_speed[True-False-True-True-False]	48.0200μs	23.8185μs	41.9842 KOps/s	41.1158 KOps/s	$+2.11\mathrm{\%}$
test_step_mdp_speed[True-False-True-False-True]	36.1510μs	20.2795μs	49.3109 KOps/s	48.1549 KOps/s	$+2.40\mathrm{\%}$
test_step_mdp_speed[True-False-True-False-False]	32.8900μs	13.0318μs	76.7355 KOps/s	74.8855 KOps/s	$+2.47\mathrm{\%}$
test_step_mdp_speed[True-False-False-True-True]	60.9610μs	38.4399μs	26.0146 KOps/s	25.5861 KOps/s	$+1.67\mathrm{\%}$
test_step_mdp_speed[True-False-False-True-False]	43.4300μs	25.4634μs	39.2721 KOps/s	38.0832 KOps/s	$+3.12\mathrm{\%}$
test_step_mdp_speed[True-False-False-False-True]	46.2000μs	22.2477μs	44.9485 KOps/s	44.1870 KOps/s	$+1.72\mathrm{\%}$
test_step_mdp_speed[True-False-False-False-False]	35.4010μs	14.9498μs	66.8905 KOps/s	65.7209 KOps/s	$+1.78\mathrm{\%}$
test_step_mdp_speed[False-True-True-True-True]	53.9210μs	36.4132μs	27.4626 KOps/s	26.7124 KOps/s	$+2.81\mathrm{\%}$
test_step_mdp_speed[False-True-True-True-False]	43.5600μs	23.6791μs	42.2314 KOps/s	40.9838 KOps/s	$+3.04\mathrm{\%}$
test_step_mdp_speed[False-True-True-False-True]	41.8910μs	24.5300μs	40.7665 KOps/s	40.0004 KOps/s	$+1.92\mathrm{\%}$
test_step_mdp_speed[False-True-True-False-False]	35.4600μs	14.9542μs	66.8707 KOps/s	65.0381 KOps/s	$+2.82\mathrm{\%}$
test_step_mdp_speed[False-True-False-True-True]	63.4310μs	38.7545μs	25.8035 KOps/s	25.4960 KOps/s	$+1.21\mathrm{\%}$
test_step_mdp_speed[False-True-False-True-False]	46.7610μs	25.6576μs	38.9748 KOps/s	38.0532 KOps/s	$+2.42\mathrm{\%}$
test_step_mdp_speed[False-True-False-False-True]	56.1410μs	26.5197μs	37.7078 KOps/s	37.4250 KOps/s	$+0.76\mathrm{\%}$
test_step_mdp_speed[False-True-False-False-False]	37.6310μs	16.8859μs	59.2209 KOps/s	58.3750 KOps/s	$+1.45\mathrm{\%}$
test_step_mdp_speed[False-False-True-True-True]	65.2310μs	40.8952μs	24.4527 KOps/s	24.2207 KOps/s	$+0.96\mathrm{\%}$
test_step_mdp_speed[False-False-True-True-False]	50.4710μs	27.5119μs	36.3479 KOps/s	35.7377 KOps/s	$+1.71\mathrm{\%}$
test_step_mdp_speed[False-False-True-False-True]	55.6810μs	26.7051μs	37.4460 KOps/s	37.5803 KOps/s	$-0.36\mathrm{\%}$
test_step_mdp_speed[False-False-True-False-False]	31.5010μs	16.9913μs	58.8536 KOps/s	57.6748 KOps/s	$+2.04\mathrm{\%}$
test_step_mdp_speed[False-False-False-True-True]	65.2610μs	42.7658μs	23.3832 KOps/s	23.3409 KOps/s	$+0.18\mathrm{\%}$
test_step_mdp_speed[False-False-False-True-False]	53.4410μs	29.5751μs	33.8123 KOps/s	33.1799 KOps/s	$+1.91\mathrm{\%}$
test_step_mdp_speed[False-False-False-False-True]	50.3210μs	28.3487μs	35.2749 KOps/s	36.0847 KOps/s	$-2.24\mathrm{\%}$
test_step_mdp_speed[False-False-False-False-False]	42.9500μs	18.6145μs	53.7216 KOps/s	52.5060 KOps/s	$+2.32\mathrm{\%}$
test_values[generalized_advantage_estimate-True-True]	25.8232ms	25.3628ms	39.4278 Ops/s	39.6450 Ops/s	$-0.55\mathrm{\%}$
test_values[vec_generalized_advantage_estimate-True-True]	82.2748ms	3.2120ms	311.3303 Ops/s	306.6637 Ops/s	$+1.52\mathrm{\%}$
test_values[td0_return_estimate-False-False]	0.1015ms	62.6042μs	15.9734 KOps/s	16.2432 KOps/s	$-1.66\mathrm{\%}$
test_values[td1_return_estimate-False-False]	54.9479ms	54.3136ms	18.4116 Ops/s	17.3651 Ops/s	$\mathbf{\text{+6.03\%}}$
test_values[vec_td1_return_estimate-False-False]	2.0385ms	1.7645ms	566.7423 Ops/s	566.5309 Ops/s	$+0.04\mathrm{\%}$
test_values[td_lambda_return_estimate-True-False]	87.0676ms	86.6928ms	11.5350 Ops/s	11.2638 Ops/s	$+2.41\mathrm{\%}$
test_values[vec_td_lambda_return_estimate-True-False]	4.0153ms	1.8056ms	553.8325 Ops/s	547.8054 Ops/s	$+1.10\mathrm{\%}$
test_gae_speed[generalized_advantage_estimate-False-1-512]	24.3834ms	24.2051ms	41.3135 Ops/s	41.3441 Ops/s	$-0.07\mathrm{\%}$
test_gae_speed[vec_generalized_advantage_estimate-True-1-512]	0.8916ms	0.7137ms	1.4011 KOps/s	1.4326 KOps/s	$-2.20\mathrm{\%}$
test_gae_speed[vec_generalized_advantage_estimate-False-1-512]	0.7124ms	0.6543ms	1.5284 KOps/s	1.5326 KOps/s	$-0.28\mathrm{\%}$
test_gae_speed[vec_generalized_advantage_estimate-True-32-512]	1.5448ms	1.4631ms	683.4590 Ops/s	685.9736 Ops/s	$-0.37\mathrm{\%}$
test_gae_speed[vec_generalized_advantage_estimate-False-32-512]	0.9190ms	0.6721ms	1.4880 KOps/s	1.4847 KOps/s	$+0.22\mathrm{\%}$
test_dqn_speed	3.9811ms	1.5153ms	659.9351 Ops/s	606.4077 Ops/s	$\mathbf{\text{+8.83\%}}$
test_ddpg_speed	3.0224ms	2.9087ms	343.7946 Ops/s	344.5404 Ops/s	$-0.22\mathrm{\%}$
test_sac_speed	8.8763ms	8.3605ms	119.6100 Ops/s	119.6616 Ops/s	$-0.04\mathrm{\%}$
test_redq_speed	11.4261ms	10.3837ms	96.3046 Ops/s	95.6712 Ops/s	$+0.66\mathrm{\%}$
test_redq_deprec_speed	11.7841ms	11.4328ms	87.4675 Ops/s	88.4215 Ops/s	$-1.08\mathrm{\%}$
test_td3_speed	8.5821ms	8.3854ms	119.2544 Ops/s	119.4362 Ops/s	$-0.15\mathrm{\%}$
test_cql_speed	26.7979ms	25.6894ms	38.9266 Ops/s	38.3712 Ops/s	$+1.45\mathrm{\%}$
test_a2c_speed	5.9066ms	5.7019ms	175.3800 Ops/s	175.5394 Ops/s	$-0.09\mathrm{\%}$
test_ppo_speed	6.9196ms	6.0251ms	165.9732 Ops/s	165.6543 Ops/s	$+0.19\mathrm{\%}$
test_reinforce_speed	4.8629ms	4.7081ms	212.3980 Ops/s	213.8605 Ops/s	$-0.68\mathrm{\%}$
test_iql_speed	20.2371ms	19.8009ms	50.5028 Ops/s	49.2080 Ops/s	$+2.63\mathrm{\%}$
test_rb_sample[TensorDictReplayBuffer-ListStorage-RandomSampler-4000]	3.0858ms	2.8800ms	347.2257 Ops/s	347.3673 Ops/s	$-0.04\mathrm{\%}$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000]	0.6826ms	0.5450ms	1.8350 KOps/s	1.8408 KOps/s	$-0.31\mathrm{\%}$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000]	4.2811ms	0.5233ms	1.9108 KOps/s	1.9243 KOps/s	$-0.70\mathrm{\%}$
test_rb_sample[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000]	3.1062ms	2.9137ms	343.2021 Ops/s	345.8518 Ops/s	$-0.77\mathrm{\%}$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000]	0.6752ms	0.5380ms	1.8587 KOps/s	1.8503 KOps/s	$+0.46\mathrm{\%}$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000]	0.1032s	0.6120ms	1.6340 KOps/s	1.9454 KOps/s	$\mathbf{\text{-16.01\%}}$
test_rb_sample[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000]	3.1004ms	3.0382ms	329.1458 Ops/s	330.4014 Ops/s	$-0.38\mathrm{\%}$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000]	0.8156ms	0.6716ms	1.4889 KOps/s	1.5030 KOps/s	$-0.94\mathrm{\%}$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000]	4.5812ms	0.6524ms	1.5327 KOps/s	1.5563 KOps/s	$-1.51\mathrm{\%}$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-RandomSampler-4000]	3.0514ms	2.8712ms	348.2918 Ops/s	348.6666 Ops/s	$-0.11\mathrm{\%}$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000]	0.6868ms	0.5443ms	1.8371 KOps/s	1.5965 KOps/s	$\mathbf{\text{+15.07\%}}$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000]	4.2118ms	0.5231ms	1.9115 KOps/s	1.9146 KOps/s	$-0.16\mathrm{\%}$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000]	3.0876ms	2.9260ms	341.7590 Ops/s	342.8442 Ops/s	$-0.32\mathrm{\%}$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000]	0.6673ms	0.5385ms	1.8570 KOps/s	1.8587 KOps/s	$-0.09\mathrm{\%}$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000]	4.3078ms	0.5216ms	1.9174 KOps/s	1.9374 KOps/s	$-1.03\mathrm{\%}$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000]	3.1037ms	3.0212ms	330.9929 Ops/s	331.0221 Ops/s	$-0.01\mathrm{\%}$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000]	1.6115ms	0.6771ms	1.4768 KOps/s	1.4996 KOps/s	$-1.52\mathrm{\%}$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000]	0.8296ms	0.6477ms	1.5440 KOps/s	1.5449 KOps/s	$-0.06\mathrm{\%}$
test_rb_populate[TensorDictReplayBuffer-ListStorage-RandomSampler-400]	0.1211s	7.0676ms	141.4898 Ops/s	142.2712 Ops/s	$-0.55\mathrm{\%}$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-400]	17.3327ms	15.1575ms	65.9739 Ops/s	65.7339 Ops/s	$+0.37\mathrm{\%}$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-400]	2.1725ms	1.0965ms	912.0195 Ops/s	908.8123 Ops/s	$+0.35\mathrm{\%}$
test_rb_populate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-400]	0.1023s	8.6429ms	115.7013 Ops/s	149.9993 Ops/s	$\mathbf{\text{-22.87\%}}$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-400]	17.5238ms	15.2296ms	65.6617 Ops/s	58.0117 Ops/s	$\mathbf{\text{+13.19\%}}$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-400]	2.1538ms	1.0804ms	925.5522 Ops/s	920.6698 Ops/s	$+0.53\mathrm{\%}$
test_rb_populate[TensorDictPrioritizedReplayBuffer-ListStorage-None-400]	0.1033s	7.1299ms	140.2552 Ops/s	141.5257 Ops/s	$-0.90\mathrm{\%}$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-400]	17.7072ms	15.5462ms	64.3243 Ops/s	63.9819 Ops/s	$+0.54\mathrm{\%}$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-400]	1.5360ms	1.4262ms	701.1719 Ops/s	696.4281 Ops/s	$+0.68\mathrm{\%}$

[vmoens]

LGTM