I am not sure what you mean by "combining" exactly.
I assume you want the parallel threads to have read and write access to all data without maintaining multiple copies.
I have dealt with a similar problem while working with large networks in a GPU environment (for doing multi-device scaling), this answer will be apt for CUDA environment, but I am sure OMP and MPI have similar mechanisms.
CUDA supports unified memory where the i/o overhead is managed at the backend. So, a simple solution is to use
cudaMallocManaged to initiate unified memory and allocate the whole matrix in it.
Now each GPU connected to the CPU can access all of the data available at this location. Naturally, lesser the amount of conflicted accesses (resulting in race conditions and page faults), better the performance. There are also some
cudaMemAdvice flags to tell the compiler if the shared data is going to be read mostly or written mostly.