Note that - as shown above - the 'Hadoop' directory and 'Spark' directory each contain a LICENSE, NOTICE, and README file. You should now have two directories and the JDK installer in your Downloads directory: Once they're extracted (Hadoop takes a while), you can delete all of the *.tar and *gz files. Note that you may need to extract twice (once to move from *gz to *.tar files, then a second time to 'untar'). Next, download 7-Zip to extract the *gz archives. Until that patched version is available (3.3.0 or 3.1.4 or 3.2.2), you must use an earlier version of Hadoop on Windows. From this point on, I'll refer generally to these versions as hadoop- and spark- please replace these with your version number throughout the rest of this tutorial.Įven though newer versions of Hadoop and Spark are currently available, there is a bug with Hadoop 3.2.1 on Windows that causes installation to fail. To avoid this, simply download from AdoptOpenJDK instead.įor Java, I download the 'Windows 圆4' version of the AdoptOpenJDK HotSpot JVM ( jdk8u232-b09) for Hadoop, the binary of v3.1.3 ( hadoop-3.1.3.tar.gz) for Spark, v3.0.0 'Pre-built for Apache Hadoop 2.7 and later' ( spark-3.0.0-preview-bin-hadoop2.7.tgz). Also, with the new Oracle licensing structure (2019+), you may need to create an Oracle account to download Java 8. Please try with Java 8 if you're having issues. I can't guarantee that this guide works with newer versions of Java. Spark seems to have trouble working with newer versions of Java, so I'm sticking with Java 8 for now: The first step is to download Java, Hadoop, and Spark.

Update : Software version numbers have been updated and the text has been clarified. I've documented here, step-by-step, how I managed to install and run this pair of Apache products directly in the Windows cmd prompt, without any need for Linux emulation. We recently got a big new server at work to run Hadoop and Spark (H/S) on for a proof-of-concept test of some software we're writing for the biopharmaceutical industry and I hit a few snags while trying to get H/S up and running on Windows Server 2016 / Windows 10. Installing and Running Hadoop and Spark on Windows

Install spark locally windows 10