Distributed Python On Multi Machine Cluster
14:20 19 May 2019

Following is the Requirement -:

class MultiMachineDoWork:

    def Function1(self, A, B):  
        return A+B

    def Function2(self, A, B):  
        return A*B 

    def Function3(self, A, B):  
        return A**B  

    def Function4():  
        X = MultiMachineDoWork.Function1(5,10)
        Y = MultiMachineDoWork.Function2(5,10)
        Z = MultiMachineDoWork.Function3(5,10)
        return X+Y+Z

Assuming that Function1,Function2 & Function3 take very long time each,its better to run them on distributed model in parallel on Machine L,M & N respectively. And Function 4 can run on Machine P which can collect the results and combine.

MapReduce Works on some sort of similar concept but runs same function on different part of Data... Can Dask / Ray / Celery be of any use in this case study...

If custom solution has to be built,what and how should the solution proceed...

Pydoop/Spark With Dask Local Cluster?


Real Life Case Study - Ensemble Model For ML Classification.One Function For RandomForest,One For Support Vector & Once For XGBoost.All running on same dataset...

python machine-learning distributed-computing