Parallel Optimizations In Scipy

I have a simple function def square(x, a=1): return [x**2 + a, 2*x] I want to minimize it over x, for several parameters a. I currently have loops that, in spirit, do somethin

Solution 1:

Here's another try, based on my original answer and the discussion that followed.

As far as I know, the scipy.optimize module is for functions with scalar or vector inputs and a scalar output, or "cost".

Since you're treating each equation as independent of the others, my best idea is to use the multiprocessing module to do the work in parallel. If the functions you're minimizing are as simple as the ones in your question, I'd say it's not worth the effort.

If the functions are more complex, and you'd like to divide the work up, try something like:

import numpy as np
from scipy import optimize
from multiprocessing import Pool

defsquare(x, a=1):
    return [np.sum(x**2 + a), 2*x]

defminimize(args):
    f,x,a = args
    res = optimize.minimize(f, x, method = 'BFGS', jac = True, args = [a])
    return res.x

# your a values
a = np.arange(1,11)

# initial guess for all the x values
x = np.empty(len(a))
x[:] = 25

args = [(square,a[i],x[i]) for i inrange(10)]
p = Pool(4)
print p.map(minimize,args)

Solution 2:

I am a bit late to the party. But this may be interesting for people who want to reduce minimization time by parallel computing:

We implemented a parallel version of scipy.optimize.minimize(method='L-BFGS-B') in the package optimparallel available on PyPI. It can speedup the optimization by evaluating the objective function and the (approximate) gradient in parallel. Here is an example:

from optimparallel import minimize_parallel
def my_square(x, a=1):
    return (x - a)**2
minimize_parallel(fun=my_square, x0=1, args=11)

Note that the parallel implementation only reduces the optimization time for objective functions with a long evaluation time (say, longer than 0.1 seconds). Here is an illustration of the possible parallel scaling:

Solution 3:

If I understand your intent, you can pass numpy arrays for both x and a, so you can optimize for all your a parameters at once.

Try something like:

defsquare(x, a=1):
    return [np.sum(x**2 + a), 2*x]

# your a values
a = np.arange(1,11)

# initial guess for all the x values
x = np.empty(len(a))
x[:] = 25# extra arguments to pass to the objective function, in this case, your a values
args = [a]

res = optimize.minimize(square, x, method = 'BFGS', jac = True, args = args)

This appears to be getting the correct results.

>>>res.x
[ -8.88178420e-16  -8.88178420e-16  -8.88178420e-16  -8.88178420e-16
  -8.88178420e-16  -8.88178420e-16  -8.88178420e-16  -8.88178420e-16
  -8.88178420e-16  -8.88178420e-16]
>>>res.fun
55.0