Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cant print Grad tensor from CTCLoss function #183

Open
adhiraj69 opened this issue Feb 23, 2021 · 0 comments
Open

Cant print Grad tensor from CTCLoss function #183

adhiraj69 opened this issue Feb 23, 2021 · 0 comments

Comments

@adhiraj69
Copy link

adhiraj69 commented Feb 23, 2021

when ever i try to print the gradient tensor obtained from gpu_ctc
i am using pytorch
and i am editing the warpctc_pytorch init.py

in the forward function i tried printing the grads tensor but i encounter the following error

i encounter this error

THCudaCheck FAIL file=/pytorch/aten/src/THC/THCCachingHostAllocator.cpp line=278 error=700 : an illegal memory access was encountered
Traceback (most recent call last):
File "CTC_loss_2.py", line 765, in
loss1 = ctcloss(x1, labels, inp1,targs1)
File "/home/dzilla69/anaconda3/lib/python3.8/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/dzilla69/anaconda3/lib/python3.8/site-packages/warpctc_pytorch-0.1-py3.8-linux-x86_64.egg/warpctc_pytorch/init.py", line 83, in forward
return self.ctc(acts, labels, act_lens, label_lens, self.size_average,
File "/home/dzilla69/anaconda3/lib/python3.8/site-packages/warpctc_pytorch-0.1-py3.8-linux-x86_64.egg/warpctc_pytorch/init.py", line 46, in forward
print(grads)
File "/home/dzilla69/anaconda3/lib/python3.8/site-packages/torch/tensor.py", line 153, in repr
return torch._tensor_str._str(self)
File "/home/dzilla69/anaconda3/lib/python3.8/site-packages/torch/_tensor_str.py", line 371, in _str
return _str_intern(self)
File "/home/dzilla69/anaconda3/lib/python3.8/site-packages/torch/_tensor_str.py", line 351, in _str_intern
tensor_str = _tensor_str(self, indent)
File "/home/dzilla69/anaconda3/lib/python3.8/site-packages/torch/_tensor_str.py", line 241, in _tensor_str
formatter = _Formatter(get_summarized_data(self) if summarize else self)
File "/home/dzilla69/anaconda3/lib/python3.8/site-packages/torch/_tensor_str.py", line 273, in get_summarized_data
return torch.stack([get_summarized_data(x) for x in (start + end)])
File "/home/dzilla69/anaconda3/lib/python3.8/site-packages/torch/_tensor_str.py", line 273, in
return torch.stack([get_summarized_data(x) for x in (start + end)])
File "/home/dzilla69/anaconda3/lib/python3.8/site-packages/torch/_tensor_str.py", line 275, in get_summarized_data
return torch.stack([get_summarized_data(x) for x in self])
File "/home/dzilla69/anaconda3/lib/python3.8/site-packages/torch/_tensor_str.py", line 275, in
return torch.stack([get_summarized_data(x) for x in self])
File "/home/dzilla69/anaconda3/lib/python3.8/site-packages/torch/_tensor_str.py", line 266, in get_summarized_data
return torch.cat((self[:PRINT_OPTS.edgeitems], self[-PRINT_OPTS.edgeitems:]))
RuntimeError: cuda runtime error (700) : an illegal memory access was encountered at /pytorch/aten/src/THC/THCCachingHostAllocator.cpp:278

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant