Pyspark: Method Isbarrier([]) Does Not Exist

I'm trying to learn Spark following some hello-word level example such as below, using pyspark. I got a 'Method isBarrier([]) does not exist' error, full error included below the c

Solution 1:

The problem is incompatibility between versions of Spark JVM libraries and PySpark. In general PySpark version has to exactly match the version of your Spark installation (while in theory matching major and minor versions should be enough, some incompatibilities in maintenance releases have been introduced in the past).

In other words Spark 2.3.3 is not compatible with PySpark 2.4.0 and you have to either upgrade Spark to 2.4.0 or downgrade PySpark to 2.3.3.

Overall PySpark is not designed to be used a standalone library. While PyPi package is a handy development tool (it is often easier to just install a package than manually extend the PYTHONPATH), for actual deployments it is better to stick with the PySpark package bundled with actual Spark deployment.

Solution 2:

Try starting your python script/session with

import findspark
findspark.init()

that updates sys.path based on the spark installation directory. Worked for me.

Solution 3:

Try to use Java 8(instead of newer versions) and also install findspark using

pip install findspark

Then try to import this at the beginning of your python script /session

import findspark
findspark.init()
from pyspark importSparkContext

This worked for me !