Netlify build error: vue-cli-service not found

I recently created a project using Vue CLI 3, and after got the project set up to a basic state I decided to add it onto Netlify and get the builds going for future reference (Netlify keeps all previous builds and if they are static sites then it also serves them at generated URLs). I started doing this late at night, and after battling it for a few hours I still didn't get anywhere. It was a pretty stressful situation, I just couldn't work out why I kept getting vue-cli-service: not found errors when Netlify runs the build. There was surprising little help online for this, maybe not many people have started using Vue CLI 3 with Netlify yet? I only managed to find out the reason after more digging and thought I'd share my solution so no one else needs to go through the same amount of pain.

The problem

Here is the full error log from Netlify:

PS: As you can see it is 2:18 am

2:21:14 AM: Build ready to start
2:21:16 AM: build-image version: 324ec043422499a87b63cac1f1dabeefe6dca19d
2:21:16 AM: build-image tag: v3.0.2
2:21:16 AM: buildbot version: 867b17c4ea5ef955db899d19b431b26c9d57e87d
2:21:16 AM: Fetching cached dependencies
2:21:16 AM: Starting to download cache of 254.8KB
2:21:16 AM: Finished downloading cache in 94.089461ms
2:21:16 AM: Starting to extract cache
2:21:16 AM: Failed to fetch cache, continuing with build
2:21:16 AM: Starting to prepare the repo for build
2:21:17 AM: No cached dependencies found. Cloning fresh repo
2:21:17 AM: git clone https://github.com/h-dong/clean-slate
2:21:18 AM: Preparing Git Reference refs/heads/master
2:21:18 AM: Starting build script
2:21:18 AM: Installing dependencies
2:21:20 AM: Downloading and installing node v8.15.1...
2:21:20 AM: Downloading https://nodejs.org/dist/v8.15.1/node-v8.15.1-linux-x64.tar.xz...
2:21:20 AM: 
                                                                           0.0%
2:21:21 AM: 
##                                                                         3.3%
2:21:21 AM: 
######################################################
2:21:21 AM:  75.9%
2:21:21 AM: 
######################################################################## 100.0%
2:21:21 AM: Computing checksum with sha256sum
2:21:21 AM: Checksums matched!
2:21:23 AM: Now using node v8.15.1 (npm v6.4.1)
2:21:24 AM: Attempting ruby version 2.3.6, read from environment
2:21:25 AM: 
2:21:25 AM: ** WARNING **
2:21:25 AM: Using custom ruby version 2.3.6, this will slow down the build.
2:21:25 AM: To ensure fast builds, set the RUBY_VERSION environment variable, or .ruby-version file, to an included ruby version.
2:21:25 AM: Included versions: 2.6.2
2:21:25 AM: 
2:21:25 AM: Required ruby-2.3.6 is not installed - installing.
2:21:26 AM: Searching for binary rubies, this might take some time.
2:21:26 AM: Found remote file https://rvm_io.global.ssl.fastly.net/binaries/ubuntu/16.04/x86_64/ruby-2.3.6.tar.bz2
2:21:26 AM: Checking requirements for ubuntu.
2:21:27 AM: Requirements installation successful.
2:21:27 AM: ruby-2.3.6 - #configure
2:21:27 AM: ruby-2.3.6 - #download
2:21:28 AM: ruby-2.3.6 - #validate archive
2:21:40 AM: ruby-2.3.6 - #extract
2:21:46 AM: ruby-2.3.6 - #validate binary
2:21:47 AM: ruby-2.3.6 - #setup
2:21:47 AM: ruby-2.3.6 - #gemset created /opt/buildhome/.rvm/gems/ruby-2.3.6@global
2:21:47 AM: ruby-2.3.6 - #importing gemset /opt/buildhome/.rvm/gemsets/global.gems
2:21:49 AM: ............................................................
2:21:49 AM: ruby-2.3.6 - #generating global wrappers
2:21:50 AM: .......
2:21:50 AM: ruby-2.3.6 - #gemset created /opt/buildhome/.rvm/gems/ruby-2.3.6
2:21:50 AM: ruby-2.3.6 - #importing gemsetfile /opt/buildhome/.rvm/gemsets/default.gems evaluated to empty gem list
2:21:50 AM: ruby-2.3.6 - #generating default wrappers
2:21:50 AM: .......
2:21:51 AM: Using /opt/buildhome/.rvm/gems/ruby-2.3.6
2:21:51 AM: Using ruby version 2.3.6
2:21:51 AM: Using PHP version 5.6
2:21:51 AM: Started restoring cached node modules
2:21:51 AM: Finished restoring cached node modules
2:21:52 AM: Installing NPM modules using NPM version 6.4.1
2:22:04 AM: > grpc@1.18.0 install /opt/build/repo/node_modules/grpc
2:22:04 AM: > node-pre-gyp install --fallback-to-build --library=static_library
2:22:05 AM: node-pre-gyp
2:22:05 AM:  WARN
2:22:05 AM:  Using needle for node-pre-gyp https download
2:22:05 AM: [grpc] Success: "/opt/build/repo/node_modules/grpc/src/node/extension_binary/node-v57-linux-x64-glibc/grpc_node.node" is installed via remote
2:22:07 AM: added 154 packages from 65 contributors and audited 41523 packages in 13.961s
2:22:07 AM: found 64 low severity vulnerabilities
2:22:07 AM:   run `npm audit fix` to fix them, or `npm audit` for details
2:22:07 AM: NPM modules installed
2:22:07 AM: Started restoring cached go cache
2:22:07 AM: Finished restoring cached go cache
2:22:07 AM: Installing Go version 1.10
2:22:12 AM: unset GOOS;
2:22:12 AM: unset GOARCH;
2:22:12 AM: export GOROOT='/opt/buildhome/.gimme_cache/versions/go1.10.linux.amd64';
2:22:12 AM: export PATH="/opt/buildhome/.gimme_cache/versions/go1.10.linux.amd64/bin:${PATH}";
2:22:12 AM: go version >&2;
2:22:12 AM: export GIMME_ENV="/opt/buildhome/.gimme_cache/env/go1.10.linux.amd64.env"
2:22:12 AM: go version go1.10 linux/amd64
2:22:12 AM: Installing missing commands
2:22:12 AM: Verify run directory
2:22:12 AM: Executing user command: npm run build
2:22:14 AM: > clean-slate@0.1.0 build /opt/build/repo
2:22:14 AM: > vue-cli-service build
2:22:14 AM: sh: 1: vue-cli-service: not found
2:22:14 AM: npm
2:22:14 AM: ERR! file sh
2:22:14 AM: npm ERR!
2:22:14 AM: code
2:22:14 AM:  ELIFECYCLE
2:22:14 AM: npm
2:22:14 AM:  ERR!
2:22:14 AM: errno ENOENT
2:22:14 AM: npm ERR!
2:22:14 AM:  syscall
2:22:14 AM:  spawn
2:22:14 AM: npm ERR! clean-slate@0.1.0 build: `vue-cli-service build`
2:22:14 AM: npm ERR! spawn ENOENT
2:22:14 AM: npm ERR!
2:22:14 AM: npm
2:22:14 AM: ERR!
2:22:14 AM:  Failed at the clean-slate@0.1.0 build script.
2:22:14 AM: npm
2:22:14 AM:  ERR!
2:22:14 AM:  This is probably not a problem with npm. There is likely additional logging output above.
2:22:15 AM: npm ERR!
2:22:15 AM:  A complete log of this run can be found in:
2:22:15 AM: npm ERR!     /opt/buildhome/.npm/_logs/2019-03-17T02_22_14_316Z-debug.log
2:22:15 AM: Caching artifacts
2:22:15 AM: Started saving node modules
2:22:15 AM: Finished saving node modules
2:22:15 AM: Started saving pip cache
2:22:15 AM: Finished saving pip cache
2:22:15 AM: Started saving emacs cask dependencies
2:22:15 AM: Finished saving emacs cask dependencies
2:22:15 AM: Started saving maven dependencies
2:22:15 AM: Finished saving maven dependencies
2:22:15 AM: Started saving boot dependencies
2:22:15 AM: Finished saving boot dependencies
2:22:15 AM: Started saving go dependencies
2:22:16 AM: Finished saving go dependencies
2:22:17 AM: Cached node version v8.15.1
2:22:17 AM: Cached ruby version 2.3.6
2:22:18 AM: Error running command: Build script returned non-zero exit code: 1
2:22:18 AM: Failing build: Failed to build site
2:22:18 AM: failed during stage 'building site': Build script returned non-zero exit code: 1
2:22:18 AM: Finished processing build request in 1m1.487616832s
2:22:18 AM: Shutting down logging, 0 messages pending

I ran the build for at least 5 times, but each time I get the same result. I tried updating dependencies, I tried adding @vue/cli as a dependency, and etc. It was late and I tried a whole bunch other "solutions" which I can't recall now. The moral of the story is that nothing worked!

I tried building locally, it works.

$ npm run build                           

⠋  Building for production...

 WARNING  Compiled with 3 warnings                                                                                                                                 03:09:34

 warning  

asset size limit: The following asset(s) exceed the recommended size limit (244 KiB).
This can impact web performance.
Assets: 
  js/chunk-vendors.29d7585c.js (716 KiB)

 warning  

entrypoint size limit: The following entrypoint(s) combined asset size exceeds the recommended limit (244 KiB). This can impact web performance.
Entrypoints:
  app (734 KiB)
      js/chunk-vendors.29d7585c.js
      css/app.8a47e692.css
      js/app.77cc03b0.js


 warning  

webpack performance recommendations: 
You can limit the size of your bundles by using import() or require.ensure to lazy load some parts of your application.
For more info visit https://webpack.js.org/guides/code-splitting/

  File                                      Size             Gzipped

  dist/js/chunk-vendors.29d7585c.js         716.30 KiB       198.37 KiB
  dist/js/app.77cc03b0.js                   11.87 KiB        4.36 KiB
  dist/service-worker.js                    0.95 KiB         0.53 KiB
  dist/precache-manifest.0eba1e4600aba3b    0.45 KiB         0.24 KiB
  67c18f1ca50ecba92.js
  dist/css/app.8a47e692.css                 6.15 KiB         1.52 KiB

  Images and other types of assets omitted.

 DONE  Build complete. The dist directory is ready to be deployed.
 INFO  Check out deployment instructions at https://cli.vuejs.org/guide/deployment.html

Hmmm... I tried building it using Travis, guess what it works!

So what is the issue then? It must be Netlify! If it weren't late in the night I totally would have spammed Netlify with some (possibly passible aggressive) messages. But I left it and went to bed instead.

The solution

The next day, after some most needed sleep and more digging around I found the issue! As expected, it was simple and stupid.

I asked Vue CLI to generate a service worker for PWA support, inside registerServiceWorker.js it uses process.env.NODE_ENV === "production" to determine if service worker should get into action (do something). Similarly, in Vuex store I am also checking process.env.NODE_ENV === "production" to decide if I should turn on strict mode or not. Naturally, I felt NODE_ENV was important and should be declared as an environment variable on Netlify to ensure the app behaves correctly.

Little did I know Netlify does something special when you add a process.env.NODE_ENV build environment variable.

If you manually set the NODE_ENV to production, any devDependencies in your package.json will NOT be installed when we automatically run yarn install or npm install at the beginning of the build.

So basically since none of my devDependencies were even getting installed, therefore the build failing should be no surprise. I just wish I found and read Netlify's documentation a bit sooner.

Source: Netlify build gochas devDependeices