Following is the Requirement -:
class MultiMachineDoWork:
def Function1(self, A, B):
return A+B
def Function2(self, A, B):
return A*B
def Function3(self, A, B):
return A**B
def Function4():
X = MultiMachineDoWork.Function1(5,10)
Y = MultiMachineDoWork.Function2(5,10)
Z = MultiMachineDoWork.Function3(5,10)
return X+Y+Z
Assuming that Function1,Function2 & Function3 take very long time each,its better to run them on distributed model in parallel on Machine L,M & N respectively. And Function 4 can run on Machine P which can collect the results and combine.
MapReduce Works on some sort of similar concept but runs same function on different part of Data... Can Dask / Ray / Celery be of any use in this case study...
If custom solution has to be built,what and how should the solution proceed...
Pydoop/Spark With Dask Local Cluster?
Real Life Case Study - Ensemble Model For ML Classification.One Function For RandomForest,One For Support Vector & Once For XGBoost.All running on same dataset...