GPU CUDA error "An unexpected error occurred trying to launch a kernel. The CUDA error was: invalid configuration argument" resolves automatically on second try

Hi,
I'm trying to run code on GPU and I get the following error:
An unexpected error occurred trying to launch a kernel. The CUDA error was:
invalid configuration argument
When I insert a breakpoint at the line that gives this error and run the code manually I get the same error but if I immediately try running it like again the code runs without the error. This proves that my code is not buggy and it is something else that's causing it to crash.
I have seen the same CUDA error using the NaN and subref functions. If it helps at all, here are the exact lines both of which give the same error on first try, but not on second try:
% On first try:
K>> W0(:,ones(1,size(dWU0,3)),:)
Error using gpuArray/subsref
An unexpected error occurred trying to launch a kernel. The CUDA error was:
invalid configuration argument
% Here it is on second try, immediately following the last one (no actions in between):
K>> W0(:,ones(1,size(dWU0,3)),:)
ans(:,:,1) =
0.0087 0.0087
0.0135 0.0135
0.0202 0.0202
0.0303 0.0303
0.0436 0.0436
0.0667 0.0667
0.0973 0.0973
0.1273 0.1273
0.1582 0.1582
0.1935 0.1935
0.2348 0.2348
0.2767 0.2767
0.3009 0.3009
0.2821 0.2821
0.2036 0.2036
0.0705 0.0705
-0.0867 -0.0867
-0.2266 -0.2266
-0.3169 -0.3169
-0.3459 -0.3459
-0.3220 -0.3220
-0.2701 -0.2701
-0.2121 -0.2121
-0.1611 -0.1611
-0.1206 -0.1206
-0.0881 -0.0881
-0.0612 -0.0612
-0.0389 -0.0389
-0.0218 -0.0218
-0.0103 -0.0103
-0.0032 -0.0032
0.0013 0.0013
0.0054 0.0054
0.0086 0.0086
0.0112 0.0112
0.0124 0.0124
0.0128 0.0128
0.0093 0.0093
0.0054 0.0054
0.0059 0.0059
0.0048 0.0048
0.0045 0.0045
0.0040 0.0040
0.0039 0.0039
0.0039 0.0039
0.0039 0.0039
0.0039 0.0039
0.0038 0.0038
0.0037 0.0037
0.0037 0.0037
0.0037 0.0037
0.0036 0.0036
0.0037 0.0037
0.0037 0.0037
0.0038 0.0038
0.0039 0.0039
0.0038 0.0038
0.0037 0.0037
0.0034 0.0034
0.0033 0.0033
0.0030 0.0030
ans(:,:,2) =
0.0620 0.0620
0.0931 0.0931
0.1302 0.1302
0.1729 0.1729
0.2109 0.2109
0.2376 0.2376
0.2496 0.2496
0.2512 0.2512
0.2390 0.2390
0.1930 0.1930
0.0968 0.0968
-0.0537 -0.0537
-0.2302 -0.2302
-0.3746 -0.3746
-0.4279 -0.4279
-0.3749 -0.3749
-0.2513 -0.2513
-0.1257 -0.1257
-0.0487 -0.0487
-0.0268 -0.0268
-0.0308 -0.0308
-0.0354 -0.0354
-0.0321 -0.0321
-0.0256 -0.0256
-0.0231 -0.0231
-0.0242 -0.0242
-0.0249 -0.0249
-0.0207 -0.0207
-0.0136 -0.0136
-0.0073 -0.0073
-0.0027 -0.0027
0.0023 0.0023
0.0103 0.0103
0.0158 0.0158
0.0197 0.0197
0.0195 0.0195
0.0170 0.0170
0.0145 0.0145
0.0147 0.0147
0.0094 0.0094
0.0094 0.0094
0.0077 0.0077
0.0048 0.0048
0.0052 0.0052
0.0047 0.0047
0.0049 0.0049
0.0048 0.0048
0.0043 0.0043
0.0033 0.0033
0.0027 0.0027
0.0026 0.0026
0.0023 0.0023
0.0025 0.0025
0.0032 0.0032
0.0044 0.0044
0.0053 0.0053
0.0060 0.0060
0.0057 0.0057
0.0045 0.0045
0.0032 0.0032
0.0016 0.0016
ans(:,:,3) =
-0.0826 -0.0826
-0.0976 -0.0976
-0.1043 -0.1043
-0.0936 -0.0936
-0.0674 -0.0674
-0.0228 -0.0228
0.0267 0.0267
0.1056 0.1056
0.1959 0.1959
0.2812 0.2812
0.3282 0.3282
0.3057 0.3057
0.2109 0.2109
0.0744 0.0744
-0.0401 -0.0401
-0.0647 -0.0647
0.0084 0.0084
0.1415 0.1415
0.2627 0.2627
0.3130 0.3130
0.2663 0.2663
0.1646 0.1646
0.0571 0.0571
-0.0321 -0.0321
-0.0928 -0.0928
-0.1287 -0.1287
-0.1545 -0.1545
-0.1726 -0.1726
-0.1818 -0.1818
-0.1817 -0.1817
-0.1754 -0.1754
-0.1644 -0.1644
-0.1537 -0.1537
-0.1374 -0.1374
-0.1179 -0.1179
-0.1006 -0.1006
-0.0818 -0.0818
-0.0607 -0.0607
-0.0417 -0.0417
-0.0325 -0.0325
-0.0220 -0.0220
-0.0142 -0.0142
-0.0041 -0.0041
-0.0028 -0.0028
-0.0012 -0.0012
-0.0020 -0.0020
-0.0022 -0.0022
-0.0012 -0.0012
0.0008 0.0008
0.0026 0.0026
0.0037 0.0037
0.0041 0.0041
0.0036 0.0036
0.0031 0.0031
0.0029 0.0029
0.0032 0.0032
0.0031 0.0031
0.0034 0.0034
0.0032 0.0032
0.0024 0.0024
0.0019 0.0019
% Here's another line of my code showing the same strange behavior:
% First try:
K>> y = nan(s,'like',x)
Error using gpuArray/nan
An unexpected error occurred trying to launch a kernel. The CUDA error was:
invalid configuration argument
% Second try:
K>> y = nan(s,'like',x)
y =
NaN
This is very perplexing to me. Any thoughts?

5 comentarios

Hi Sia,
Can I get the following information which would be helpful in finding the issue.
  1. Which Driver version you are using?
  2. Which toolkit and it version if you are using one.
  3. The sizes of the data structures you are using.
  4. Whether you are using GPUarray or GPUCoder
  5. If you could share the script /function that would give more insight.
Hi Sia, I had the same issue running kilosort2. Did you figure out a fix in the end? I saw something similar was posted as an issue to kilosort2's github, but Marius closed the issue due to inactivity.
I have the same problem when using permute with gpuArray matrix.
Have there been any solutions? Running into the issue with kilosort 2.
I'm also getting this issue, but only for certain recordings.

Iniciar sesión para comentar.

Respuestas (1)

Maybe you can consider trying to reduce the size of W0. I encountered this problem before, and it won't happen again until I reduce the size.

Categorías

Productos

Etiquetas

Preguntada:

Sia
el 19 de Jul. de 2019

Respondida:

Ben
el 5 de Ag. de 2021

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by