Google Cloud Machine Learning Out Of Memory

I am having a issue of getting out of memory when I choose the following configuration (config.yaml): trainingInput: scaleTier: CUSTOM masterType: large_model workerType: co

Solution 1:

I have fixed the problem. I needed to do few things:

1. Change the tensorflow version especially how I was submitting training job in cloud.

2. Instead of one hot encoding [which creates a column for each new item being added], I switched to Feature Hashing

Now it can train a categorical dataset with 2.5 millon rows and 4200 encoded columns.