PyTorch 1.5 adds C++ power, distributed training. news. ... The RPC framework is designed to minimize the amount of data copying across nodes, so work is always done as close to the data as possible.

Soumith Chintala, PyTorch project lead, seems to share Zaharia's ideas about distributed training being the next big thing in deep learning, as it has been introduced in the latest version of PyTorch.