NeuroEvolution using TensorFlow JS - Part 2

May 27 2019

Introduction

This tutorial is part 2, if you have not completed NeuroEvolution using TensorFlowJS - Part 1, I highly recommend you do that first. As it will:

• Explain how to setup the codebase
• How to run the local dev environment
• How to setup the basic NeruoEvolution implementation

In this tutorial we will be:

• Adding visual debugging information
• Adding more inputs for TensorFlow to use to predict jumping
• Implement crossover within TensorFlowJS
• Use the best progressing NeuralNetwork to ensure faster NeuroEvolution
• Add saving and loading of NeuralNetwork models

This tutorial is also based on the source code found on my GitHub account here https://github.com/dionbeetson/neuroevolution-experiment.

Let's get started!

Add in visual/debugging

We will be adding in useful debugging information to the right hand column and to the game itself.

Chart the progress of each evolution

We are going to use ChartJS to track the progress (as a percentage) and the score for the best game of each evolution. Create a file called js/ai/NeuroEvolutionChart.js and copy the below logic into it. It is a basic ChartJS implementation that I won't go into detail in this tutorial.

js/ai/NeuroEvolutionChart.js

class NeuroEvolutionChart {
  constructor() {
    this.chart = null;
    this.config = {
    type: 'line',
    data: {
     labels: [],
     datasets: [{
      label: '',
      backgroundColor: '#999999',
      borderColor: '#EEEEEE',
          borderWidth: 3,
      data: [],
      fill: false,
     },{
      label: '',
      backgroundColor: '#91C7B1',
      borderColor: '#A1CFBC',
          borderWidth: 2,
      data: [],
      fill: false,
     }]
    },
    options: {
     responsive: true,
     title: {
      display: false
     },
     tooltips: {
      mode: 'index',
      intersect: false,
     },
        legend: {
          display: false
        },
     hover: {
      mode: 'nearest',
      intersect: true
     },
     scales: {
      xAxes: [{
       display: false,
       scaleLabel: {
        display: true,
        labelString: 'Round'
       }
      }],
      yAxes: [{
       display: true,
            ticks: {
              beginAtZero: true,
              min: 0,
              max: 100
            },
       scaleLabel: {
        display: false,
        labelString: 'Progress'
       }
      },{
       display: true,
            position: 'right',
            ticks: {
              beginAtZero: true,
              min: 0,
              max: 200
            },
       scaleLabel: {
        display: false,
        labelString: 'Score'
       }
      }]
     }
    }
   };

    this.setup();
  }

  setup(){
    var ctx = document.getElementById('trend').getContext('2d');
    this.chart = new Chart(ctx, this.config);
  }
}

Now, add an update() method that will be responsible for pushing the data points onto the chart. It will also only keep 12 data points (the very first evolution, and the latest 11 evolutions). This is to avoid clutter on the page.

js/ai/NeuroEvolutionChart.js

update(progress, score) {
  // Purge middle data (to avoid clutter)
  if( this.config.data.labels.length > 12 ) {
    this.config.data.labels.splice(1, 1);
    this.config.data.datasets[0].data.splice(1, 1);
    this.config.data.datasets[1].data.splice(1, 1);
  }

  this.config.data.labels.push((this.config.data.labels.length+1))
  this.config.data.datasets[0].data.push(progress);
  this.config.data.datasets[1].data.push(score);

  this.chart.update();
}

Back in the NeuroEvolution class add the following variable declaration to the top

js/ai/NeuroEvolution.js

#neuroEvolutionChart = new NeuroEvolutionChart();

Then in the same file in the finishGeneration(games, timeTaken) method, right after the line this.#bestGames.sort(this.sortByFitness); add in.

js/ai/NeuroEvolution.js

// Update UI - Chart
this.#neuroEvolutionChart.update(bestPlayerByFitness.progress, bestPlayerByFitness.score);

Reload your browser, and click the 'start evolution' button. After each evolution you will start to see the chart in the top right update.

Additional debugging visuals

Let's quickly add in some more useful debug information which includes:

• The best game progress to date
• Track the best score from each evolution
• A progress bar in the header to track round and generation progress

In the NeuroEvolution class add the below methods

js/ai/NeuroEvolution.js

updateUIaddBestGenerationToBestScore( pickBestPlayerByFitness, timeTaken ) {
  let bestScore = document.createElement("li");
  bestScore.innerHTML = pickBestPlayerByFitness.score + ' (' + pickBestPlayerByFitness.progress.toFixed(1) + '%) (' + pickBestPlayerByFitness.fitness.toFixed(3) + ') (' + timeTaken + 's)';
  this.#bestScores.insertBefore(bestScore, document.querySelector("li:first-child"));
}

updateUIRoundInformation() {
  document.querySelector("#round-current").innerHTML = this.#generation;
  document.querySelector("#round-total").innerHTML = this.#maxGenerations;
  document.querySelector("#round-progress").style.width = '0%';
  document.querySelector("#generation-progress").style.width = (this.#generation/this.#maxGenerations)*100 + '%';
}

updateUIBestPlayerScore( bestGame ) {
  document.querySelector("#best-player-score").innerHTML = bestGame.score + " points (" + bestGame.progress.toFixed(1) + "%)";
}

In the first line of the start() method add:

js/ai/NeuroEvolution.js

processGeneration(games, bestPlayerBrainsByFitness) {
  this.updateUIRoundInformation();
  ...

Then right after where you added in this.#neuroEvolutionChart.update... add in:

js/ai/NeuroEvolution.js

// Update UI
this.updateUIaddBestGenerationToBestScore(bestPlayerByFitness, timeTaken);
this.updateUIBestPlayerScore(this.#bestGames[0]);
this.updateUIRoundInformation();

Lastly, add the following variable declaration to the top

js/ai/NeuroEvolution.js

#bestScores = null;

and the following constructor

js/ai/NeuroEvolution.js

constructor() {
  this.#bestScores = document.querySelector("#best-scores");
}

Reload your browser, and click the 'start evolution' button. You should start seeing debug information in your right hand column.

Another useful debug is to visualise the inputs that are being used to predict the jump() action.

Back in the Ai class within the think() method, just before the game.brain.predict() line, add in this code block to push debug points into the game canvas.

js/ai/Ai.js

game.gameApi.setDebugPoints([
  {
    x: 0,
    y: inputs[0]
  },
  {
    x: inputs[2],
    y: 0
  },
  {
    x: 0,
    y: inputs[3]
  },
  {
   x: 0,
   y: inputs[4]
  }
]);

Reload your browser, and click the 'start evolution' button. You should start seeing crosshairs for the inputs being used by TensorFlow JS on every prediction call.

Sometimes it is useful to be able to 'pause the game before the next evolution begins. You will see a checkbox in the left column labelled 'Pause after generation'. Let's wire it up, as it currently is not.

In ui.js add in the below change handler to capture user interaction.

js/ai/ui.js

document.querySelector("#ml-pause-before-next-generation").addEventListener("change", function() {
  neuroEvolution.pauseBeforeNextGeneration = this.checked;

  if( false == neuroEvolution.pauseBeforeNextGeneration && neuroEvolution.pausedGames.length > 0 && neuroEvolution.pausedBestNeuralNetworksByFitness.length > 0 ) {
    neuroEvolution.start(neuroEvolution.pausedGames, neuroEvolution.pausedBestNeuralNetworksByFitness);
  }
});

neuroEvolution.pauseBeforeNextGeneration = document.querySelector("#ml-pause-before-next-generation").checked;

This isn't enough though, we need to build in the logic to actually pause the game. We are basically going to wrap the ai.start(bestPlayerBrainsByFitness); call within the start() method in NeuroEvolution.js in an IF statement based on if the pause checkbox is checked. If it is checked, we will then store the current games within a local class variable so we can retrieve it once the checkbox is unchecked.

The completed method looks like this now.

js/ai/NeuroEvolution.js

start(games, bestPlayerBrainsByFitness) {
  this.updateUIRoundInformation();

  if ( this.#generation < this.#maxGenerations ) {

    if( false == this.#pauseBeforeNextGeneration ){
      for ( let i = 0; i < games.length; i++ ) {
        games[i].gameApi.remove();
      }

      games = undefined;

      this.#pausedGames = [];
      this.#pausedBestNeuralNetworksByFitness = [];

      this.#generation++;

      const ai = new Ai(this.finishGeneration.bind(this));
      ai.start(this.#useImageRecognition, bestPlayerBrainsByFitness);

    } else {
      this.#pausedGames = games;
      this.#pausedBestNeuralNetworksByFitness = bestPlayerBrainsByFitness;

      for ( let i = 0; i < games.length; i++ ) {
        games[i].gameApi.show();
      }
    }
  } else {
    this.enableSpeedInput();
  }
}

Then add in the following variable declarations in the NeuroEvolution class.

js/ai/NeuroEvolution.js

#pausedGames = [];
#pausedBestNeuralNetworksByFitness = [];
#pauseBeforeNextGeneration = false;

And then add in the required getters/setters in the NeuroEvolution class.

js/ai/NeuroEvolution.js

get pauseBeforeNextGeneration() {
  return this.#pauseBeforeNextGeneration;
}

set pauseBeforeNextGeneration( pauseBeforeNextGeneration ) {
  this.#pauseBeforeNextGeneration = pauseBeforeNextGeneration;
}

get pausedGames() {
  return this.#pausedGames;
}

get pausedBestNeuralNetworksByFitness() {
  return this.#pausedBestNeuralNetworksByFitness;
}

get bestGames() {
  return this.#bestGames;
}

set pauseBeforeNextGeneration( pauseBeforeNextGeneration ) {
  this.#pauseBeforeNextGeneration = pauseBeforeNextGeneration;
}

Speeding up NeuroEvolution

Ok that is debugging done for now. Let's move onto improving our NeuroEvolution to solve this game a little faster. This involves 3 things:

1. Adding more inputs for TensorFlow to use to predict jumping
2. After each evolution randomly choosing to use the 'best game to date', as this can keep things on track if we have a single bad evolution randomly (trust me it happens)
3. Implementing a crossover method so instead of mutating only 1 of the best games, we take 2 of the best games and breed a child NeuralNetwork from their 2 results (which is called a crossover)
4. If a game passes the level, don't mutate or crossover, as we already have a successful NeuralNetwork

Adding more inputs for TensorFlow JS

In the Ai class we are going to add in 3 more inputs for (isPlayerJumping, playerVelocity & canPlayerJump). Within the think() method after the existing 5 inputs, add in these additional inputs.

js/ai/Ai.js

// Is player jumping
inputs[5] = (game.gameApi.isPlayerJumping() ? 1 : 0);
inputsNormalised[5] = map(inputs[5], 0, 1, 0, 1);

// Player velocity
inputs[6] = (game.gameApi.getPlayerVelocity() ? 1 : 0);
inputsNormalised[6] = map(inputs[6], -1.1, 1.1, 0, 1);

// Can play jump?
inputs[7] = (game.gameApi.canPlayerJump() ? 1 : 0);
inputsNormalised[7] = map(inputs[7], 0, 1, 0, 1);

Then change the variable declaration for inputs in Ai class to 8

js/ai/Ai.js

#inputs = 8; 

Implement crossover within TensorFlowJS

In the NeuroEvolution class within the completeGeneration() method, we are going to modify the breeding section to make it a little smarter. Make the for loop look like this.

js/ai/NeuroEvolution.js

// Breeding
for (let i = 0; i < games.length; i++) {
  let bestPlayerA = this.pickBestGameFromFitnessPool(games);
  let bestPlayerB = this.pickBestGameFromFitnessPool(games);
  let child;

  child = this.mutateNeuralNetwork(this.crossoverNeuralNetwork(bestPlayerA.neuralNetwork.clone(), bestPlayerB.neuralNetwork.clone()));
 
  bestPlayerBrainsByFitness.push(child);
}

Now we need to add in the crossover method to the same file.

js/ai/NeuroEvolution.js

crossoverNeuralNetwork(neuralNetworkOne, neuralNetworkTwo) {
  let parentA_in_dna = neuralNetworkOne.input_weights.dataSync();
  let parentA_out_dna = neuralNetworkOne.output_weights.dataSync();
  let parentB_in_dna = neuralNetworkTwo.input_weights.dataSync();
  let parentB_out_dna = neuralNetworkTwo.output_weights.dataSync();

  let mid = Math.floor(Math.random() * parentA_in_dna.length);
  let child_in_dna = [...parentA_in_dna.slice(0, mid), ...parentB_in_dna.slice(mid, parentB_in_dna.length)];
  let child_out_dna = [...parentA_out_dna.slice(0, mid), ...parentB_out_dna.slice(mid, parentB_out_dna.length)];

  let child = neuralNetworkOne.clone();
  let input_shape = neuralNetworkOne.input_weights.shape;
  let output_shape = neuralNetworkOne.output_weights.shape;

  child.dispose();

  child.input_weights = tf.tensor(child_in_dna, input_shape);
  child.output_weights = tf.tensor(child_out_dna, output_shape);

  return child;
}

Reload your browser, and click the 'start evolution' button. You should start seeing games solve themselves a bit faster in fewer generations (normally around 20).

10% chance to crossover best game

Back in the NeuroEvolution class, within breeding section, let's add one final update so that on random occasions (10% chance), we do a crossover with the best game to date. Make the breeding section look like this

js/ai/NeuroEvolution.js

// Breeding
for (let i = 0; i < games.length; i++) {
  let bestPlayerA = this.pickBestGameFromFitnessPool(games);
  let bestPlayerB = this.pickBestGameFromFitnessPool(games);
  let bestPlayerC = this.#bestGames[0];
  let child;

  if ( random(1) < 0.1) {
    const ai = new Ai();
    let bestPlayerD = new NeuralNetwork(ai.inputs, ai.neurons, ai.outputs);
    child = this.mutateNeuralNetwork(this.crossoverNeuralNetwork(bestPlayerC.neuralNetwork.clone(), bestPlayerD));
  } else {
    child = this.mutateNeuralNetwork(this.crossoverNeuralNetwork(bestPlayerA.neuralNetwork.clone(), bestPlayerB.neuralNetwork.clone()));
  }

  bestPlayerBrainsByFitness.push(child);
}

We also need to expose getters in the Ai class for the input, neuron & output variables. Add the following methods:

js/ai/Ai.js

get inputs(){
  return this.#inputs;
}

get neurons(){
  return this.#neurons;
}

get outputs(){
  return this.#outputs;
}

Reload your browser, and click the 'start evolution' button. You should start seeing games solve themselves a bit faster in fewer generations (normally around 10-15).

Don't mutate on success

Now wrap the Breed functionality in an IF statement to ensure that if a game passes the level, we don't mutate or crossover, we just continue using the successful NeuralNetwork.

js/ai/NeuroEvolution.js

if ( false != gamePassedLevel ) {
  for ( let i = 0; i < games.length; i++ ) {
    if (games[i].gameApi.isLevelPassed() ) {
      games[i].neuralNetwork.save('neuralNetwork');
      for (let ii = 0; ii < games.length; ii++) {
          bestPlayerBrainsByFitness.push(games[i].neuralNetwork.clone());
        }
      }
    }

    console.log('Level Passed:', this.#bestGames[0], this.#bestGames.length, this.#bestGames);
    this.start(games, bestPlayerBrainsByFitness);
  } else {
    // Breeding
    for (let i = 0; i < games.length; i++) {
      let bestPlayerA = this.pickBestGameFromFitnessPool(games);
      let bestPlayerB = this.pickBestGameFromFitnessPool(games);
      let bestPlayerC = this.#bestGames[0];
      let child;

      if ( random(1) < 0.1) {
        const ai = new Ai();
        let bestPlayerD = new NeuralNetwork(ai.inputs, ai.neurons, ai.outputs);
        child = this.mutateNeuralNetwork(this.crossoverNeuralNetwork(bestPlayerC.neuralNetwork.clone(), bestPlayerD));
      } else {
        child = this.mutateNeuralNetwork(this.crossoverNeuralNetwork(bestPlayerA.neuralNetwork.clone(), bestPlayerB.neuralNetwork.clone()));
      }

      bestPlayerBrainsByFitness.push(child);
    }
 
   this.start(games, bestPlayerBrainsByFitness);
}

Reload your browser, and click the 'start evolution' button. Once enough evolutions process to pass the level, you will notice that all future games in evolutions will start passing consistently.

This is a far as we will take this tutorial in regards to evolving and speeding up our NeuroEvolution logic. Next we will move onto saving/loading our NeuralNetwork models.

Saving & Loading our NeuralNetwork models

The last piece to implement is the saving and loading functionality of our NeuralNetwork. This is so once you have a NeuralNetwork that works you can save it and load it back when needed. We will be implementing the:

1. Ability to automatically save a NeuralNetwork that passes to localStorage
2. Ability to manually save best NeuralNetwork to localStorage
3. Ability to manually save best NeuralNetwork to disk
4. Ability to load from local storage
5. Ability to load from disk

First we need to add the ability to save within the NeuralNetwork.js file. Add these 2 methods

js/ai/NeuralNetwork.js

stringify() {
  let neuralNetworkToSave = {
    input_weights: this.input_weights.arraySync(),
    output_weights: this.output_weights.arraySync()
  };

  return JSON.stringify(neuralNetworkToSave);
}

save( key ) {
  localStorage.setItem(key, this.stringify());
}

Automatically save a NeuralNetwork that passes to localStorage

In the NeuroEvolution. class within the method finishGeneration(). Add in the below to the IF statement when a game level is passed.

js/ai/NeuroEvolution.js

games[i].neuralNetwork.save('neuralNetwork');

So it should look like this:

js/ai/NeuroEvolution.js

for ( let i = 0; i < games.length; i++ ) {
  if (games[i].gameApi.isLevelPassed() ) {
    games[i].neuralNetwork.save('neuralNetwork');
    for (let ii = 0; ii < games.length; ii++) {
      bestPlayerBrainsByFitness.push(games[i].neuralNetwork.clone());
    }
  }
}

console.log('Level Passed:', this.#bestGames[0], this.#bestGames.length, this.#bestGames);
this.start(games, bestPlayerBrainsByFitness);

Your game will now automatically save the NeuralNetwork model to localStorage key neuralNetwork if it passes the level.

Manually save best NeuralNetwork to localStorage

To manually save to local storage add the below code to ui.js

js/ai/ui.js

document.querySelector("#btn-ml-save-neuralnetwork-localstorage").addEventListener('click', function (event) {
  if( neuroEvolution.bestGames.length > 0 ) {
    neuroEvolution.bestGames[0].neuralNetwork.save('neuralNetwork')
  }
});

Manually save best NeuralNetwork to disk

To manually save to disk you can add the below code to ui.js

js/ai/ui.js

document.querySelector("#btn-ml-save-neuralnetwork-disk").addEventListener('click', function (event) {
  if( neuroEvolution.bestGames.length > 0 ) {
    let neuralNetworkJSON = neuroEvolution.bestGames[0].neuralNetwork.stringify();
    let a = document.createElement("a");
    let file = new Blob([neuralNetworkJSON], {type: 'application/json'});
    a.href = URL.createObjectURL(file);
    a.download = 'neuralNetwork.json';
    a.click();
  }
});

Loading NeuralNetwork Models

To load from local storage or disk we need to create a method to handle loading a NeuralNetwork into memory. Create the below method in ui.js (there is probably a better file location for this method now that I think about it).

js/ai/ui.js

const loadNeuralNetwork = (neuralNetworkData) => {
  const ai = new Ai();
  let neuralNetwork = new NeuralNetwork(ai.inputs, ai.neurons, ai.outputs);
  neuralNetwork.dispose();

  neuralNetwork.input_weights = tf.tensor(neuralNetworkData.input_weights);
  neuralNetwork.output_weights = tf.tensor(neuralNetworkData.output_weights);

  let neuralNetworks = [];
  for ( let i = 0; i < ai.totalGames; i++ ) {
    neuralNetworks.push(neuralNetwork);
  }

  let games = neuroEvolution.pausedGames;
  neuroEvolution.pausedGames;
  neuroEvolution.reset();

  neuroEvolution.start(games, neuralNetworks);
};

We also need to add the below method into the Ai class.

js/ai/Ai.js

get totalGames(){
  return this.#totalGames;
}

Load from local storage

Now to load from localStorage into memory, add the below into ui.js

js/ai/ui.js

document.querySelector("#btn-ml-load-neuralnetwork-localstorage").addEventListener('click', function (event) {
  cachedLevelSections = [];
  let games = neuroEvolution.pausedGames;
  neuroEvolution.pausedGames;
  neuroEvolution.reset();

  let neuralNetworkData = localStorage.getItem('neuralNetwork');

  loadNeuralNetwork(JSON.parse(neuralNetworkData));
});

We also need to add the reset method into the NeuroEvolution class.

js/ai/NeuroEvolution.js

reset() {
  this.#bestGames = [];
  this.#generation = 1;
  this.#pausedGames = [];
  this.#pausedBestNeuralNetworksByFitness = [];
  this.#pauseBeforeNextGeneration = false;
}

Load from disk

Now to load from disk into memory, add the below into ui.js

js/ai/ui.js

document.querySelector("#btn-ml-load-neuralnetwork-disk").addEventListener('change', function (event) {
  let file = this.files[0];
  let reader = new FileReader()
  let textFile = /application\/json/;
  let fileText = '';

  if (file.type.match(textFile)) {
    reader.onload = function (event) {
       let importedNeuralNetwork = JSON.parse(event.target.result);

       loadNeuralNetwork(importedNeuralNetwork);
    }
  }

  reader.readAsText(file);
});

Reload your browser, and click the 'start evolution' button. You will notice if you inspect Chrome DevTools you will see a localStorage key called 'brain' once a level is passed.

You can also now also use the buttons in the left hand menu to save and load the best NeuralNetwork to disk.

Within the GitHub codebase within the models/ directory you will see 3 existing models that can be loaded up to pass levels 1, 2 & 3.

Try it yourself and get your NeuroEvolution implementation to pass Level 1, then Level 2, then finally Level 3.

I get consistent results along the lines of:

• Level 1: Takes 5-20 generations
• Level 2: Takes 5-20 generations
• Level 3: Takes 40-700 generations (as it has to learn to jump blocks and gaps).

Congratulations!

I hope this tutorial was useful for you to learn how to use TensorFlowJS to build a NeuroEvolution implementation. Any questions leave them in the comments below, or tweet me on twitter at @dionbeetson

Source code

All source code for this tutorial can be found here https://github.com/dionbeetson/neuroevolution-experiment.

Part 3 - coming soon...

Coming soon and will be using image recognition to solve the same browser based game.

Read More

NeuroEvolution using TensorFlowJS - Part 1

May 26 2019

Introduction

This tutorial will walk you through how to implement NeuroEvolution using TensorFlow JS. This NeuralNetwork will learn how to play a simple browser based game that requires a player to jump over blocks and gaps. It is based on the source code found on my GitHub account here https://github.com/dionbeetson/neuroevolution-experiment.

The difference in this solution compared to other solutions out there is that instead of building ML directly into the core game source code (like I saw in a lot of examples); the ML and Game source code are completely decoupled. This simulates a much more realistic real world example, as you would rarely have direct access to the source code when applying ML (you would only have access to specific inputs). I created an an API/SDK for the browser based game to expose some key functionality (the inputs the NeuralNetwork needs), but end state is to use image analysis so this solution is more adaptable to other applications.

References

Although I wrote this codebase myself I took a lot of input/influence from a huge range of great tutorials that already exist. You can see a list of the key tutorials/repos I learned from on the GitHub Readme for this project here.

Let's get started!

Setup

Clone this repo from GitHub:

git clone git@github.com:dionbeetson/neuroevolution-experiment.git

Then run in terminal the below to setup the repository for this tutorial:

npm install

npm run setup-tutorial

This basically moves the js/ai directory to js/ai-source, and leaves everything else in place. If you ever need to refer back to the original source code use js/ai-source as reference.

Now run the below to startup the app:

npm run dev

The browser will load up the application, but you will see 404 errors within the DevTools console. You will though however, have a working browser game. Click 'Play game', and use the spacebar to jump to test things out. Throughout this tutorial you are going to be building a NeuroEvolution implementation to learn how to automatically solve this game.

Quick overview of the UI

• Left section: Includes base controls for Manual and AI interactions. In order from top to bottom
• Level: You can select the level (there are currently 3 levels available in the game in order of complexity)
• Play game: Play an user interacted game
• Enable drawing: Whether you want to render the drawing on canvas
• Speed: How fast the game will run
• Start Evolution: Start the NeuroEvolution
• Save to localStorage: Save best current NeuralNetwork into localStorage
• Save to disk: Save best current NeuralNetwork to disk
• Load from localStorage: Load current NeuralNetwork in local storage into memory and start NeuroEvolution
• Load from disk: Load a NeuralNetwork from disk into memory and start NeuroEvolution
• Pause after generation: For debugging, pause after each evolution so you can see / debug it
• Middle section: The game(s) render within the large middle column
• Right section: Is used to display debugging information for the NeuroEvolution implementation

This tutorial has 4 key files

For the entire tutorial we will be working within the js/ai/ directory.

1. ui.js: Controls the UI interactions
2. NeuralNetwork.js: Is the base neural network
3. Ai.js: This class instantiates (by default 9) games & corresponding NeuralNetwork instances
4. Neuroevolution.js: This class will run up to 1500 generations of Ai.js until it successfully passes the selected level

Setup base structure

Let's start to build out enough base logic to be able to predict if we should jump or not using TensorFlowJS.

Create a directory js/ai, and create a file called js/ai/ui.js which will handle basic UI controls. Within this file we want to instantiate the NeuroEvolution class (that we are about to create, and bind a UI button to start the evolution process).

js/ai/ui.js

let neuroEvolution = new NeuroEvolution();

document.querySelector("#btn-ml-start").addEventListener('click', function (event) {
  neuroEvolution.start([]);
}, false);

Now we need to create the skeleton of NeuralNetwork.js, Ai.js & Neuroevolution.js.

For the NeuralNetwork let's just start with a constructor that allows us to send input_node/hidden_node & output_node count arguments.

js/ai/NeuralNetwork.js

class NeuralNetwork {
  constructor(input_nodes, hidden_nodes, output_nodes) {
    // The amount of inputs (eg: player y position, height of next block etc..)
    this.input_nodes = input_nodes;
    // Amount of hidden nodes within the Neural Network)
    this.hidden_nodes = hidden_nodes;
    // The amount of outputs, we will use 2 (will be needed for level 3)
    this.output_nodes = output_nodes;

    // Initialize random weights
    this.input_weights = tf.randomNormal([this.input_nodes, this.hidden_nodes]);
    this.output_weights = tf.randomNormal([this.hidden_nodes, this.output_nodes]);
  }
}

Now we need to create the Ai class. This will include a method that will create 1 game for now and link it with a NeuralNetwork instance. We will start with a start() method to create instances of a NeuralNetwork, a checkGame() method that will be called every 50ms which will call the think() method that we will expand to make the actual prediction to jump or not jump.

js/ai/Ai.js

class Ai {
  #totalGames = 1; // Will increase to 9 games later
  #inputs = 5;
  #neurons = 40;
  #outputs = 2;
  #games = [];
  #gamesRunning = 0;
  #sectionsToSeeAhead = 1;
  #timeTakenDateStart = null;

  start(useImageRecognition, neuralNetworks, completeCallback) {
    this.#timeTakenDateStart = new Date();

    for ( let i = 0; i < this.#totalGames; i++ ) {
      let neuralNetwork;

      if ( undefined !== neuralNetworks && neuralNetworks[i] instanceof NeuralNetwork ) {
        neuralNetwork = neuralNetworks[i];
      } else {
        neuralNetwork = new NeuralNetwork(this.#inputs, this.#neurons, this.#outputs);
      }

      let gameApi;

      if ( useImageRecognition ) {
        gameApi = new GameImageRecognition();
      } else {
        gameApi = new GameApi();
      }

      this.#games[i] = {
        gameApi: gameApi,
        neuralNetwork: neuralNetwork,
        interval: null
      }

      // Debug look ahead
      this.#games[i].gameApi.setHighlightSectionAhead(this.#sectionsToSeeAhead)

      // Start game
      this.#gamesRunning++;
      this.#games[i].gameApi.start();

      this.#games[i].interval = setInterval(this.checkGame.bind(null, this, this.#games, this.#games[i]), 50);
    }
  }

  checkGame(ai, games, game) {
    if( game.gameApi.isSetup() ) {
      ai.think(game);
    }
  }

  think(game) {
  }
}

Now create the NeuroEvolution class that will be responsible for running multiple evolutions of Ai.js until we get an evolution that passes the level.

js/ai/NeuroEvolution.js

class NeuroEvolution {
  #generation = 1;
  #maxGenerations = 1500;
  #useImageRecognition = false;

  start(games, bestPlayerBrainsByFitness) {

    if ( this.#generation < this.#maxGenerations ) {

      for ( let i = 0; i < games.length; i++ ) {
        games[i].gameApi.remove();
      }

      games = undefined;

      this.#generation++;

      const ai = new Ai(this.finishGeneration.bind(this));
      ai.start(this.#useImageRecognition, bestPlayerBrainsByFitness);
    } else {
      this.enableSpeedInput();
    }
  }

  finishGeneration(games, timeTaken) {
   // TODO
  }

  updateUIRoundInformation() {
    document.querySelector("#round-current").innerHTML = this.#generation;
    document.querySelector("#round-total").innerHTML = this.#maxGenerations;
    document.querySelector("#round-progress").style.width = '0%';
    document.querySelector("#generation-progress").style.width = (this.#generation/this.#maxGenerations)*100 + '%';
  }
}

Now you have your base structure in place.

Reload your browser (http://127.0.0.1:8080/), and click the 'Start evolution' button. You should see 1 game playing, and failing (no jumping), no matter how many times you reload the page. That is because we haven't implemented the prediction logic. Let's implement that using TensorFlowJS in a moment.

Quickly, before that, in the Ai class, let's create 9 games at once instead of 1, as NeuroEvolution will be incredibly slow with only 1 NeuralNetwork at a time. Depending on your computer specs you can increase this to a higher number. I did find that 9 worked quite well though.

js/ai/Ai.js

#totalGames = 9

Reload your browser, and click 'Start evolution' button again. You should see 9 games playing.

Predicting whether to jump or not

To use TensorflowJS to predict to jump, we need to modify a few files. First in Ai.js in the think() method add the below logic in. Essentially we are accessing public information that the JS game exposes in js/game/GameAPI.js (like player position, next block height etc..) and then normalising all inputs to be between 0-1 as TensorFlow JS requires this.

js/ai/Ai.js

think(game) {
  let inputs = [];
  let inputsNormalised = [];

  // Player y
  inputs[0] = (game.gameApi.getPlayerY());
  inputsNormalised[0] = map(inputs[0], 0, game.gameApi.getHeight(), 0, 1);

  // Player x
  inputs[1] = game.gameApi.getPlayerX();
  inputsNormalised[1] = map(inputs[1], inputs[1], game.gameApi.getWidth(), 0, 1);

  let section = game.gameApi.getSectionFromPlayer(this.#sectionsToSeeAhead);

  // 2nd closest section x
  inputs[2] = section.x + section.width;
  inputsNormalised[2] = map(inputs[2], inputs[1], game.gameApi.getWidth(), 0, 1);

  // 2nd closest section y
  inputs[3] = section.y;
  inputsNormalised[3] = map(inputs[3], 0, game.gameApi.getHeight(), 0, 1);

  // 2nd closest section y base
  inputs[4] = section.y + section.height;
  inputsNormalised[4] = map(inputs[4], 0, game.gameApi.getHeight(), 0, 1);

  // Call the predict function in the NeuralNetwork
  let outputs = game.neuralNetwork.predict(inputsNormalised);

  // If input is about 0.5 then jump, if not, stay still
  if ( outputs[0] > 0.5 || outputs[1] > 0.5 ) {
    game.gameApi.jump();
  }
}

We also will need to create the predict() method in the NeuralNetwork class. This method creates an input_layer, hidden_layer and output_layer and returns 2 predictions (which are both numbers between 0-1).

js/ai/NeuralNetwork.js

predict(user_input) {
  let output;
  tf.tidy(() => {
    let input_layer = tf.tensor(user_input, [1, this.input_nodes]);
    let hidden_layer = input_layer.matMul(this.input_weights).sigmoid();
    let output_layer = hidden_layer.matMul(this.output_weights).sigmoid();
    output = output_layer.dataSync();
  });
  return output;
}

Reload your browser, and click the 'start evolution' button.

You will now see 9 games playing, and some should be jumping, and some not. This is completely random. Reload a few times if you need to to ensure this is the behaviour you are seeing.

Once all of your players die you will notice the game stops. As of now, we have used TensorFlow JS to make a prediction about whether to jump or not. This is based on a set of inputs (player X, player Y & section ahead height/Y).

Now we need to add in the NeuroEvolution aspect so that we pick the best progressed games and start a new evolution with those NeuralNetworks (but we will evolve/mutate them slightly before hand).

In NeuroEvolution add a method called calculateFitness(). This method iterates through all games once finished, creates a fitness rating which is just the % progress of the game. It then has logic to increase the fitness of better performing games, so that later on in our logic, there is a higher chance the better performing NeuralNetworks are chosen for the next evolution. Hope that all makes sense.

js/ai/NeuroEvolution.js

calculateFitness(games) {
  for ( let i = 0; i < games.length; i++ ) {
    let game = games[i];
    games[i].fitness = game.gameApi.getProgress() / 100;
    games[i].score = game.gameApi.getScore();
    games[i].progress = game.gameApi.getProgress();
  }

  // The below code makes the better progressed games have a higher fitness so they have a higher chance of being selected for next generation
  games.sort(this.sortByFitness);
  games.reverse();

  let prev = 0;
  for ( let i = 0; i < games.length; i++ ) {
    games[i].fitness = this.#discountRate * prev + games[i].fitness;
    prev = games[i].fitness;
  }

  games.sort(this.sortByFitness);

  return games;
}

In NeuroEvolution.js add a method that will check if at least 1 game has passed the level in the current evolution.

js/ai/NeuroEvolution.js

didAtLeastOneGameCompleteLevel(games) {
  for ( let i = 0; i < games.length; i++ ) {
    if (games[i].gameApi.isLevelPassed() ) {
      return games[i];
    }
  }

  return false;
}

In NeuroEvolution add a method that will pick the exact best player with the highest fitness. We use this to display and keep a track of the best performing NeuralNetwork.

js/ai/NeuroEvolution.js

pickBestGameByActualFitness(games){
  let game;
  let prevFitness = 0;
  for ( let i = 0; i < games.length; i++ ) {
    if (games[i].fitness > prevFitness) {
      game = games[i];
      prevFitness = game.fitness;
    }
  }

  return game;
}

In NeuroEvolution add a method that will pick one (not always the best like the above method) of the best players with the highest fitness

js/ai/NeuroEvolution.js

pickBestGameFromFitnessPool(games) {
  let index = 0;
  let r = random(1);

  while (r > 0 ) {
    if( undefined !== games[index] ) {
      r = r - games[index].fitness;
      index++;
    } else {
      r = 0;
    }
  }
  index--;

  let game = games[index];

  return game;
}

In NeuroEvolution we can now improve the finishGeneration() method with the below logic. This code will:

1. Calculate fitness for all 9 finished games
2. Store the best player in a local variable
3. Add the best player to the bestGames[] array
4. Ensure we only keep the best 5 games (for memory/performance reasons)
5. Iterate over all games and mutate the best performing NeuralNetworks into new NeuralNetworks for the next evolution
6. Start the next NeuroEvolution

js/ai/NeuroEvolution.js

finishGeneration(games, timeTaken) {
  games = this.calculateFitness(games);

  // Did one of the games finish?
  let gamePassedLevel = this.didAtLeastOneGameCompleteLevel(games);

  let bestPlayerByFitness = gamePassedLevel;
  let bestPlayerBrainsByFitness = [];

  if( false === bestPlayerByFitness ){
    bestPlayerByFitness = this.pickBestGameByActualFitness(games);
  }

  this.#bestGames.push(bestPlayerByFitness);
  this.#bestGames.sort(this.sortByFitness);

  // Only keep top 5 best scores
  if( this.#bestGames.length > 5 ) {
    this.#bestGames = this.#bestGames.slice(0, 5);
  }

  // Breeding
  for (let i = 0; i < games.length; i++) {
    let bestPlayerA = this.pickBestGameFromFitnessPool(games);
    let child;

    child = this.mutateNeuralNetwork(bestPlayerA.neuralNetwork.clone());

    bestPlayerBrainsByFitness.push(child);
  }

  this.start(games, bestPlayerBrainsByFitness);
}

Lastly, in NeuroEvolution you will need to add the below variable declarations.

js/ai/NeuroEvolution.js

  #discountRate = 0.95;
  #bestGames = [];

Reload your browser, and click the 'start evolution' button. You should again see 9 games playing, some will also be jumping like last time. However, it still wont progress past the first evolution. This is because we need to wire up Ai.js to call the callback method in the checkGame method and add some logic to check if the game is over. Let's update the checkGame() method to look like the below.

js/ai/Ai.js

checkGame(ai, games, game) {
  if( game.gameApi.isOver() ) {
    clearInterval(game.interval);

    ai.#gamesRunning--;
    document.querySelector("#round-progress").style.width = ((games.length-ai.#gamesRunning)/games.length)*100 + '%';

    if( ai.areAllGamesOver(games) && 0 == ai.#gamesRunning ) {
      let timeTakenDateComplete = new Date();
      let timeTaken = (timeTakenDateComplete - ai.#timeTakenDateStart) / 1000;

      ai.#completeCallback(games, timeTaken);
    }
  } else {
    if( game.gameApi.isSetup() ) {
      ai.think(game);
    }
  }
}

We also need to add the following variable declarations to Ai class.

js/ai/Ai.js

#completeCallback;

Reload your browser, and click the 'start evolution' button.

You should see an error as we didn't declare the method areAllGamesOver(). Let's quickly add the below to Ai

js/ai/Ai.js

areAllGamesOver(games) {
  for ( let i = 0; i < this.#totalGames; i++ ) {
    if( false == games[i].gameApi.isOver() ) {
      return false;
    }
  }

  return true;
}

... Another error, we forgot to set the completeCallback. In Ai, create a constructor up the top after the variable declaration.

js/ai/Ai.js

constructor(completeCallback) {
  this.#completeCallback = completeCallback;
}

Reload your browser, and click the 'start evolution' button.

Ok, we are making progress, but we are now missing the clone() method. We need to be able to clone NeuralNetworks after each generation so we don't interfere with previous NeuralNetwork objects.

In NeuralNetwork class add the clone() method as well as the dispose() method, as that is called from within the clone() method.

js/ai/NeuralNetwork.js

clone() {
  return tf.tidy(() => {
    let clonie = new NeuralNetwork(this.input_nodes, this.hidden_nodes, this.output_nodes);
    clonie.dispose();
    clonie.input_weights = tf.clone(this.input_weights);
    clonie.output_weights = tf.clone(this.output_weights);
    return clonie;
  });
}

dispose() {
  this.input_weights.dispose();
  this.output_weights.dispose();
}

Reload your browser, and click the 'start evolution' button. You will see we are close now, we are just missing the mutateNeuralNetwork() method. This method is responsible for slightly tweaking the cloned NeuralNetwork so that you get a slightly different result in the next evolution. Let's go ahead and add the mutateNeuralNetwork() method into the NeuroEvolution class. I'm not going to go into more detail on this, but you can google it and find this method in a few other examples out there. I also wasn't the original author of this, I just adjusted it to work for my implementation. See GitHub for reference to original source code.

js/ai/NeuralEvolution.js

mutateNeuralNetwork(b) {
  function fn(x) {
    if (random(1) < 0.05) {
      let offset = randomGaussian() * 0.5;
      let newx = x + offset;
      return newx;
    }
    return x;
  }

  let neuralNetwork = b.clone();
  let ih = neuralNetwork.input_weights.dataSync().map(fn);
  let ih_shape = neuralNetwork.input_weights.shape;
  neuralNetwork.input_weights.dispose();
  neuralNetwork.input_weights = tf.tensor(ih, ih_shape);

  let ho = neuralNetwork.output_weights.dataSync().map(fn);
  let ho_shape = neuralNetwork.output_weights.shape;
  neuralNetwork.output_weights.dispose();
  neuralNetwork.output_weights = tf.tensor(ho, ho_shape);
  return neuralNetwork;
}

Reload your browser, and click the 'start evolution' button. Now after each of the 9 games fail the first time, the UI will start a new evolution. If you let this go for a few minutes you should see the game get solved - it normally takes around 30-50 generations with this current implementation.

Congratulations!

This is a very simple implementation of NeuroEvolution. In Part 2, we will go through making improvements to really speed things up including:

• Implementing crossover within TensorFlowJS
• Using the best progressing NeuralNetwork to ensure faster NeuroEvolution
• Saving and loading models
• Adding debugging information

Source code

All source code for this tutorial can be found here https://github.com/dionbeetson/neuroevolution-experiment.

Part 2

Continue on to NeuroEvolution using TensorFlowJS - Part 2

Read More

Talking at WordCamp Europe 2016 about News Corp Australia's migration to WordPress VIP

October 30 2016 At the start of 2016 myself and Juan (a friend and colleague) decided it would be a great idea to present on NewsCorp Australia's migration to WordPress at WordCamp Europe in Vienna. We had been through a huge migration project over the previous 18 month, moving Australia's largest media websites over to an enterprise version of WordPress.

Jump forward 2 months and we found out we were accepted, followed by 2 months of planning and cramming 18 months of project work into a 30 minute presentation. Before you know it we were flying out to Vienna from Sydney Australia for a quick 3 day trip at WordCamp Europe - the largest WordCamp event in the world.

Below is the talk we presented and transcript on how we scaled WordPress to support over 3 million posts across 22 sites, with over 16 million users a day.

Transcript

00:00

18 months ago, News Corp Australia, started its journey migrating all of its websites over to WordPres. Both Juan here and myself were on the software engineering team part of that migration project. We had often heard at the start of this project, WordPress doesn't scale. Well our talk today is to prove them wrong, and to demonstrate that WordPress provides a really good base. It's a really solid platform, and if you implement the correct technical solutions to extend WordPress, it can scale really well for large and high volume traffic websites like ours.

00:48

Historically people may have interpreted News Corp Australia to be a little like this, a 90 year old company, potentially focusing a lot on their newspaper side of the business. But over the past 18 months, a few of the people you can see in this photo plus a lot more in News Corp, migrated 22 of Australia largest sites including news.com.au, theaustralian.com.au & foxsports.com.au over to WordPress VIP. We've imported over 3 millions posts, and are receiving around 16 million users a day.

01:26

Why did we move to WordPress VIP? First of all, what is WordPress VIP? WordPress VIP is a hosting option for WordPress sites that is powered by wordpress.com which is an enterprise level hosting. We needed to escape our old CMS, which was unscalable and very expensive to maintain. WordPress was chosen because it is a very reliable tool, and most of our journalists are already familiar with, making training and up skilling across thousands of journalists much easier. We just needed to find a way to scale WordPress to meet the needs of our business.

02:06

Lastly and possibly the best part, with WordPress VIP we didn't need to worry about this. wordpress.com manages our production infrastructure, so updates to the core get directly deployed, we don't need to worry about updating core, or server patches. Our main focus is developing the best new features for our customers.

At a high level we have developed 45 WordPress plugins and 22 themes. apart from our production infrastructure, we run 58 AWS non production environments, which are a combination of EC2, RDS and SQS, where we try to mimic as much as technically possible the Wordpress.com environments. From a deployment perspective, we run 25 deployments to production each week through our CI/CD platform; and we love our unit tests, we have over 5000 and counting.

03:21

Our journey started in November 2014, when we look back at those 18 months, there are 5 key factors that really challenged us. We wish we had of known about these before we started the project. however, encountering these challenges and discovering their solutions were part of our journey, and actually a great team building exercise. I'm not going to lie though, it was a tough tough journey.

We wanted to share our story for 2 main reasons. First, to demonstrate that WordPress can scale for the enterprise, and secondly, to share the challenges we faced and the solutions we implemented.

4:03

We are going to focus on 5 key areas; site build; front-end; content ingestion: authoring - how we are creating and updating new stories; as well as continuous integration and deployments. We are going to talk about the challenges we faced and the solutions we implemented.

4:24

The first talking point, side build. We really needed the most simplistic way for our WordPress admins to manage and organise the content on our sites. Customizer was a great start to this, it provided a really good base, but we needed to extend it, and this is the functionality we are going to take you through. Working really closely with the XWP team who I can see a lot of in the front here. we added a lot of functionality into customizer, a fair bit of which has been moved into WordPress core. Here is a short video demonstrating just some of the few key features that have been added in or may already now exist in WordPress core.

05:02

All of our sites utilize multiple content areas - that is pretty standard stuff. Within each of our content areas, we have this concept of default widgets, or also known as global widgets, which will render on every single page. We are currently on the business page, and we are looking at the right hand rail, it's just a list of stories. If we now go to the news section in the navigation, you will notice that the widgets we just saw and actually the same widgets that were just rendered. So we can spin up a lot of pages that have very similar structure very quickly. Then to extend that, there is a plugin which we developed which allows you to override the widgets on a specific page and content area. We are on the homepage now, and by checking the 'localise to the current page' checkbox, it allows your to add a completely different set of new widgets for a specific content area for that specific page.

05:54

We call these contextual settings and they are stored as a JSON data structure as a custom post type in WordPress. The next thing you are going to see is our homepage, and that is the next big challenge we had. We have a lot of widgets that keep scrolling, and scrolling, and it's maybe a bit too much content, and that's what caused us a few challenges. By default WordPress as I'm sure a few of us know, stores all of its widget data in sidebars which is then stored in WordPress options. Because we have a huge huge homepage, it meant that the amount of data we were storing in WordPress options was more than 1mb. We leverage Memcache quite extensively, and 1mb is unfortunately the limit in Memcache, so as soon as any key has more than 1mb of data, it's not longer getting cached. This meant that for every single page request, we weren't caching WordPress options, which was a huge performance issue for us.

Another plugin was developed, we called this widgets-plus. Essentially it's a new custom post type and it stores all of the widget data now in its own custom post type in WordPress. This means that we can store everything relevant to just a single widget in isolation, and cache it in isolation, we don't need to worry about caching everything as part of WordPress options. Within our previous CMS it took us hours, sometimes a day to build a relatively simple page. within WordPress, we only showed you a very small part of it, but we can build really complex pages in literally a few minutes. It's a huge efficiency boost for our business.

07:24

After a few months we had a solution for site build, now the front-end architecture was really key for us. We needed to make sure when all of the sites were moving across into WordPress, we weren't going to end up with a mess of spaghetti front-end code.

07:38

I'm sure all of you here like us who are developers (front-end or backend) know that PHP as a templating language, isn't necessarily the best approach. It gets really messy really quickly. We needed a templaitng engine to ensure we decoupled our front-end code from our backend. Twig was our templating engine of choice, and there are 2 plugins that power this implementation. The first one is called vip-twig, and this one is responsible for integrating Twig into WordPress core - it is a very standalone plugin. Our next plugin, template-integration is responsible for exposing all of those core WordPress functions into our twig templates. It basically bridges the gap between WordPress core and the data our templates need to render out posts on the front-end. The good part about this is if we need to implement a new templating language because twig is no longer the laungauge to use, we can keep our template-integration plugin as it's all reusable, and we can simply build a new vip-smarty or vip-ejs plugin to manage that integration for that new templating engine.

08:40

We have now built our sites, and we have a templating engine in place, now we need content. We have been working on content ingestion functionality for around 6 months in parallel, and in May 2015, we were moving into beta testing in a VIP production environment, planning on launching our first site in June 2015. As you can see from our slides, the timeline is no where near the far right of the screen - yes we ran into some problems.

09:14

In our first attempt we ran into 2 major problems. First our posts are authored in a different platform, they are not authored in WordPress, and we had a benchmark of 60 seconds that content needed to be published into the WordPress front-end from the moment it was saved. Secondly we were seeing some rare race conditions in content ingestion, which caused content to be duplicated on our WordPress sites.

As well as the above 2 major problems, when we moved into production we needed to do an initial content import of around 300,000 posts per site. We didn't have MySQL access as our sites are hosted on wordpress.com - we couldn't expect them to give us access. It was a challenge to import that amount of information. Lastly, after that we, would have a constant stream of data updates including news stories, videos, photos coming into our sites at a rate of around 16 updates per second.

10:28

We needed an architecture to support this. Here is a simplified diagram (it is a very simplified diagram) of what we have implemented. It can be read from bottom to top. First our journalist will create a story or update it in one of our editorial tools. This post will then be sent to our API platform which manages all of our content. It will organise it, categorise it, and make it available to other platforms to consume it, WordPress being one of those platforms. On top of that, as soon as a message arrives into the API platform, it will send a notification to our SQS system in WordPress informing it of the new story and to 'start' ingesting.

11:18

With the help of the WordPress VIP team, we developed an ingestor daemon, we call it Turbo, because in the beginning it was built using WordPress crons, but wp-crons were not meant to work in that way. Like I said before, we were doing 16 updates a second and wp-cron were not coping with that, they were not fast enough. We needed to create this multi-threaded daemon. This whole process that you are seeing is what we call 'end to end publishing'. With this architecture in place, we have a single entry point for content imports, whether we are doing 300,000 updates in an initial import, or 16 updates a second. We also have with SQS, reduced the amount of duplicate content, actually it is now at zero, and finally we can ingest content in less than 60 seconds, we are basically ingesting content in less than 10 seconds, as soon as a journalist saves a story, it appears on our site within about 6 seconds.

12:38

Another challenge that we faced was the amount of fields and data we had that represented a 'NewsStory', or a 'WordPress post in our business. This is stored in our APIs, this is the object we get. As you can see there is a trimmed down version of how our JSON object looks, it's massive, it's a really big object. At the beginning we thought lets split this into pieces, save everything into post meta fields, and build them up again to send writes back to our APIs. However, we thought, why re-invent the wheel, if we are already using this JSON object across all our other platforms, let's just save it in our WordPress database, and consume it from there, and read it from there. Due to this we decided to store this JSON object in the post_content field in the posts table. With this architecture, we needed to change how our templates work, and how our plugins work with this field. We are no longer reading it as text field, but as a JSON object.

Saving the JSON object here allowed us to have revisions of an entire post object. Also when rendering a post on the page, we can do it with a single database call. We finally kept a simple rule, we would only use post_meta for searchable parameters.

All this gave us a huge performance benefits by more than halving the amount of database calls.

14:19

As you can probably guess, June came and past and we didn't launch our sites. But they stayed in beta testing for a few months. Our internal development team, the XWP team, and WordPress VIP team, worked for around 2 months to implement solutions to all the problems we just discussed; and one night in September, a very very late night, we launched our first site on WordPress VIP. It was a night filled with anxiety, way too much pizza, and then a sigh of relief when it switched on and everything just worked.

By now we have implemented a simplistic site build approach, our front-end is clean, and we have content quickly ingesting into our WordPress sites.

15:21

Now we needed to use WordPress for what it is meant to be used for; for editing and creating content. We developed these authoring screens within WordPress that expose our data JSON structure that defines our News story content so people can edit it in the most simplistic way. As you can see we leveraged meta boxes quite extensively, and used CSS and Javascript to provide the look and feel we were after. We tried not to deviate from WordPress core at all so any updates that come through to WordPress.com did not affect us.

15:58

Here is another image of one of our authoring screens to demonstrate what we are doing for image management within posts. We are leveraging off the media element in WordPress and allowing authors to edit/crop images within the UI.

16:15

Our category management is also a little different, our stories have to have in the permalink, the URL of the specific category they belong to. So we needed to add in a primary category (or section as we call it), that they have to select when creating a story. Then you can put that story in multiple categories if needed.

16:48

Within the first week of February in 2016, we launched our last 4 sites into wordpress.com in just one week. And everything just rolled out very very smoothly. Then gradually over the next 4 months, until pretty much last week, all that authoring functionality that Juan just showed you, rolled out into a production environment.

17:08

So we needed to maintain this complex WordPress platform that we had setup. We also needed to ensure that at all costs it was impossible for any cowboy coders to rise up and try and take control of our codebase.

For this it was really critical that we had a solid continuous integration and deployment platform setup. We needed to make sure that a developer can get his code or her code from their local environment all the way into production in the most stable and risk free way.

17:31

For us it comes down to feature based branching and deployments. Every feature for sits within it's own feature branch, branched off master in GIT. It's always developed in isolation, it's always tested in isolation, it's always deployed in isolation. Our team chose to never bundle up more than one feature and deploy it at once, as for us this added too many risks and too many dependencies.

Our team then utilizes pull requests to control any code merges into master. This is where code reviews happen, which is one of our favourite pastimes. Every single line of code is code reviewed by another developer. No single line of code reaches production without being code reviewed by at least 1, 2 or sometimes 3 other developers in the team.

18:14

We then utilize Atlassians Bamboo continuous integration platform to manage all of our automation. This is where on every code commit we run PHPCS to make sure all of our coding standards are being met, we run PHPUnit to make sure all of our unit tests are passing and there are no regression issues we are aware of. Once all of those tests pass we package up the application, running any front-end build tasks in gulp or grunt, we compile our TWIG templates, then we deploy that out to our 50+ non-production environments.

This is when testing happens, either manual regression testing, or we have a suite of selenium based automation tests using the robot framework. Once all manual and automated tests have passed we deploy this feature through our continuous integration platform all the way into the WordPress VIP environment. We celebrate with a few beers, and then we start that process all over again for the next feature.

The best part is, this isn't the process we started with 18 months ago. This evolved over countless retrospectives, team huddles, and big issues with our previous processes, to what it is today.

19:48

With a solid continuous integration and deployment platform setup, the only other challenge we faced was with WordPress VIP you cannot control the exact time your code is deployed to production, and if your feature has changes in 3, 4 or 5 different code repositories, you can't guarantee the order the code in those repositories are always deployed in.

19:32

We needed a way to be able to turn features on and off in a production environment. Welcome to the feature toggle,  This was developed by one of our developers called Roman back in Australia, and it is an implementation of the feature toggle technique by Martin Fowler. It abstracts all of the complexities around feature toggling. So with just a few lines of code you can setup a new feature, you can give it a nice descriptive name, and in the last parameter (the boolean) you can decide if it is active or deactivated by default.

20:00

Then all of our themes and plugins have access to check if this feature is enabled or not. If it is enabled you can execute a certain chunk of code. Lastly this plugin exposes an admin page in WordPress, that lists all of the features we currently have available to us, it's de-active and activated state, and this is where you can toggle a certain feature on or off. This feature toggle has allowed us to, for our really big features, to roll them out to all of our sites from a deployment perspective, and then gradually toggle them on one site at a time so we can stagger the regression testing required.

20:32

So we have come to the end, and what have we really learnt? First of all, as Juan pointed out, migration projects are hard hard work. No matter matter how much planning you do, you are always going to run into unknowns. For us it was really critical for us to get our code into a beta version on the WordPress VIP infrastructure, so we could test our end to end solutions in their environment, rather than our non-production setup.

To all those people that said to us that WordPress isn't going to scale for what we are doing. Well if you just download WordPress core, install it on a single server an try and support 16 million uses, then well no it isn't going to work. But if you implement the correct solution architecture, with things like memcache, elastic search, asynchronous content ingestion, it scales really well for sites like ours.

21:21

Lastly, us partnering with two really awesome teams. With XWP who really put a lot of effort in at the start of our project, their expertise in WordPress was absolutely priceless for us. They were very pivotal in our project, especially in the first 12 months, and a lot of us have become really good mates.

Secondly, with the WordPress VIP team, they just kept going and going just like the energizer bunny, they never stopped no matter what challenges and issues we threw their way.

21:58

At at very high level that was our teams journey to WordPress over the last 18 months. If anyone has any questions I believe we have some time now otherwise please come find us throughout WordCamp in Vienna if you’re chasing any more details on our talk, or just want to have a chat.

Read More

Experimenting with mob programming at NewsCorp Australia

April 8th 2016
Mob programming has been gathering more and more momentum over the last 6-12 months so we decided to experiment with it in some of our software development teams at NewsCorp Australia. So what exactly is Mob Programming?

”The basic concept of mob programming is simple: the entire team works as a team together on one task at the time. That is: one team – one (active) keyboard – one screen (projector of course). It’s just like doing full-team pair programming.”

Why mob programming?

How often do specific developers in a team only understand certain aspects of the code? And while code reviews are a great collaboration technique, sometimes you need more. Mob programming is a great technique to solve technical problems together as a team, allowing everyone's input to be voiced. As a positive side affect it can also help to encourage pair programming within a development team.

So how did mob programming go in NewsCorp?

Like anything you rarely find success by forcing a particular practice onto a team, it needs to adopted by a team naturally if they see it really adds value. We started by simply running a few exercises with a few of our front-end and back-end development teams to experiment on how mob programming could work within our business.

Front-end development team

 

Our session had 9 developers (rather large for a single session), all taking part in a JavaScript FizzBuzz challenge. Each developer was driving for a single 3 minute interval (we only had time to go around once).

Opening remarks started with, "ahh Fizzbuzz, this should be easy", and "I'm not sure how I feel about people watching me code".

Round 1: & 2:

Mood: Cautious and careful
It started off fast as the first person to drive took the laptop and started coding.. until... the team quickly pointed out, they have to listen to the navigators. This is when everything slowed down and everyone started discussing, "do we do unit tests first", "can we re-read the instructions in the readme", and "this is harder than you think when we all have to do it together". By the end of the first round, the team had created the start of the "FizzBuzz" method.

By the end of the second round the basic FizzBuzz method was complete.

Round 3:

Mood:  Focused and more opinionated
Seeing the team started with the core functionality, at the start of round 3, questions were being raised about starting unit tests, and how they were going to test this functionality. By the end of round 3, we had an initial unit test that tested the basic 'FizzBuzz' was outputting correctly.

Round 4-5:

Mood: Positive and excited
Round 4 and 5 saw a wider set of unit tests being created to test the validity of Fizz, Buzz and FizzBuzz outputs.

Round 6:

Mood: Calm
There was some refactoring of the FizzBuzz application, as well as some cleanup to the unit tests to improve coverage.

Round 7-8:

Mood: Frantic and fun
By the last 2 rounds the team focused on outputting all of the numbers from 1-100 in the console. Challenge complete!

The teams observations:

The team started off slow in giving directions, well, in the way the driver understood the instructions being given. It took a few rounds for the team members to be talking the same language. After this, all someone had to say was, "created a unit test", "copy that function and past it there", "rename this to that".

As expected, there were the more confident developers navigating more than others at the start, but when it was their turn they immediately became the listeners, which gave an opportunity for other developers to voice up.

The mood was also really positive. At the start people were saying, "oh now everyone sees how I code in real time", but by the end everyone realized their peers were quite open and not judgemental.

Back-end development team:

This session had 5 developers, all taking part in the same FizzBuzz challenge (but in PHP). As before, each developer was driving for a single 3 minute interval, however we went around 2 rotations.

Opening remarks started with, "ok, lets see how this goes"

Round 1, 2 & 3:

Mood: Energetic
Team was keen to get started quickly and smashed through most of the requirements in about 2-3 rounds.

Round 4, 5 & 6:

Mood: Strategic and careful
These rounds saw the team work closely on unit testing to get confidence in the core code that was developed. A lot of discussion around correct architecture also took place, when the team decided what and how to refactor.

Round 7, 8 & 9:

Mood: Agreement, disagreement and focus
Within the last 3 rounds, the team rapidly implemented refactoring to allow better unit testing of the implementation of FizzBuzz.

Observations:

Withing 3 rounds the application was done. It was a quick MVP to get the basics in place. This development team had been working together for over 12 months so they knew how to work with each other very well.

The team took the approach of fleshing out an MVP, then adding unit tests, then refactoring the application to improve scalability. Why did they not do unit testing first? This is a team that has thousands of unit tests around their current code base, and there is a culture that code never reaches production without unit tests. To them, getting a quick MVP with the understanding that unit tests will be there for release is already well accepted.

What the team did uncover, was using sessions like this to flesh out complex features together could really speed up the overall delivery of a future feature. Currently code reviews are heavily used in this team, and this is when potential issues in design decisions are picked up. But having a 1-2 hour session to mob out the basic architecture of a new feature could really reduce feedback in code review process.

What's next? Will we keep mob programming?

This is up for each team to decide... I believe we all agree, sessions like this can be great to get an entire team over a single concept or feature, or to architect a new application with everyones input right from the start.

It was definitely a fun exercise that we will keep experimenting with.

Read More

Making code reviews part of your team culture

March 14th 2016

Code reviews are like unit testing, as developers, we really want to do them, but not every team is able to introduce them successfully. This is a post around a presentation I recently gave at What Do You Know Sydney about how to make code reviews part of your development team culture.

I want to share how code reviews in one of the teams NewsCorp are no longer a manual chore, but are a part of the team culture and are treated in the same way as scoping, development and testing.

I've worked with amazing development leads over my career who dedicate so many hours each and every night code reviewing and providing great feedback on the code developers in the team have developed. Although this works and adds huge value in ensuring the highest level of code, it is not a scalable solution.

What is a code review?

“Using your experience as a developer to review and assume joint ownership of that code”

What do code reviews involve?

• Checking out the code locally and ensuring it runs as expected.
• Making sure the code complies to the teams syntax standards.
• Running the unit tests (and ensuring there are unit tests).
• Making sure the code architecture & design patterns being used are the right ones.
• Being 100% certain there are no security vulnerabilities in the code.
• ...Every team will have it's own interpretation of what a code review entails...

Why do code reviews fail within some teams?

• It's an ad-hoc process.
• Developers don't understand what a code review should involve.
• There is not enough time allocated to code reviews.
• When only the leads or senior developers are doing the code reviews.
• Most importantly, when there are no surrounding processes in place to encourage code reviews.

How do you make code reviews part of your development team culture?

Make it part of your task estimation

• Unit testing is usually around 20% of the development time (on average), we have found that code reviews are around 5% of the development time. If a feature takes a 5 day week to develop, usually half a day is enough to code review this newly developed feature. 
• Remember to allow some time to implement development feedback from the code review.

Utilize Pull Requests (and enforce approvals before merging)

In their simplest form, pull requests are a mechanism for a developer to notify team members that they have completed a feature. In the open source community this is normally how new changes are submitted to core contributors to review and implement into the core open source platform. Within your team ensuring that at least one developer must code review and approve a Pull Request before it is merged is a great step to encouraging code reviews within a team.

Use a great tool like Stash for code reviews

• Stash is a great tool from Atlassian that makes code reviews simple.
• You can configure stash to enforce an approver for every code review before it can be merged. 
• You can actually take it a step further and not allow developers to merge code (only allow testers to merge Pull Requests, this allows the testing team to control the flow of code that goes into UAT and production environments). 
• Stash has an awesome UI to make comments on code reviews easy and to even allow adding tasks to Pull Requests that have to be completed before merging.

Add it to your Agile board

• Don't hide the code review process - keep it out in the open and display it on your Agile board.
• This encourages the wider business to accept code reviews as part of the day-to-day.
• If a tasks fails code review it moves back into development.

Use code reviews as a knowledge sharing tool

This is the single greatest value code reviews have given the team. I often hear in development teams that this developer knows this, and that developer knows that. When your team embraces code reviews, you obviously still have people experienced more in different areas of the code, but at least 2 developers understand the same area of the code. This is because 1 developer has coded it, and at least 1 other developer has reviewed it - they have checked it out locally, ran the unit tests, understood the code and gave their tick of approval for it to be released.

How can you introduce code reviews in your team?

• Don't ask permission just do it!
  As developers it is our responsibility to own building stable, secure and scalable code. If we need to spend time unit testing and code reviewing our code, then that is our responsibility to do so (and fight for it).
• Use a great tool to encourage code reviews like Stash.
• Use Pull Requests within your branching strategy and enforce approvals.
• Don’t have exemptions - it's all or nothing!!
• Have a reward each sprint or month for the person who does the most amount of code reviews
• Have a token for code review immunity if you find some developers are getting too many code reviews.
  We use a Jedi robe, if you are the one wearing the Jedi robe (of which there is only 1), then no person can assign you a code review until you take off the robe.
• Most importantly, be an honest code reviewer.
  
  Don't just approve a Pull Request, every constructive piece of feedback you add, is making your whole team stronger and better developers.

The team I am referring to in this presentation has code reviewed  over 172,000 lines of code in the last 12 months which in my books is an impressive statistic.

Here is a copy of the presentation I gave at What Do You Know Sydney.

Read More

How to efficiently build an MVP for your tech startup

It's surprising how often I meet entrepreneurs and start-ups who have been building their tech startup for more than a year (sometimes up to 2 and half years) and they haven't launched yet. When working on a startup you often want to make sure it's perfect before you launch so you can show your customers how amazing your product is. But what happens if they don't like what you have built? How can you not waste years building a product that your customers might not even want?

Words like 'Lean Engineering', 'MVP' & 'Agile development' have become mainstream within the tech startup industry in the last few years. Without getting too technical (but following these practices) here are some tips to stay on track, stay focused and get your MVP out to your customers FAST (within 2-3 months)!

What is an MVP?

MVP stands for "Minimal Viable Product". It refers to what is the simplest product you can design, build & release to your customers with minimum time and cost investment. Think about what will give you the best ROI or validation that what your building is what your customers "actually" want?

MVP is sometimes also referred to as a 'Minimum Feature Set' by some people. Be careful though, an MVP is not a broken featureless site, it is a carefully constructed 'prototype' that is a subset of your large product goal that you can build and release very quickly to "validate" your idea. The image below demonstrates really well how the MVP process should work.

Wan't to read more about MVP's? Check out this article.

Your goal

Your goal is to build the smallest, useable subset of your product that can be used to test against your 'real customers' to determine if you are building a product they 'actually' want to use (and more importantly, pay for!).

What you want to DO (to deliver fast)

• Scope and define your MVP requirements (what features are required to launch your MVP)?
• Stick to your requirements.
• Define & manage your priorities (on a day-to-day basis with your team).
• Create 2 TODO/backlog lists:
• "List 1" is your MVP TODO list (these are must have features for launch).
• "List 2" is your "After MVP" launch TODO list (these are all the cool features you come up with that aren't critical for launch and can be done after your MVP is released).
• Define your timeframe.
• 2-3 months is your ideal timeframe to complete design, development, testing & release.
• Try to really stick to a 3 month maximum timeframe for your MVP.
• Collaborate together as much as possible (share an office space).
• If you collaborate together you will get more done together.
• Have morning standup meetings.
• Standup meetings are part of 'Agile project management' and are quick informal catchups where everyone defines what they did yesterday and what they are doing today. They are around 5-10 minutes, everybody must standup, and it usually occurs at the start of the day.
• Always have a TODO list every day about what you and your team are going to get done
• Have stop limits that will stop you spending too much time on unimportant tasks.
• Stick to your requirements.
• Don't add new features unless they really block your initial MVP launch.
• If you really need to add a new feature (or someone in the team is pushing for a new feature):
1. Ask your team, is this feature mission critical for launch?
• If YES, take out another feature that is less important.
• If NO, put it in the technical backlog for after launch.
• Don't rewrite your MVP on new technology stacks unless you physically cannot launch your MVP without doing so.
• Be doers, not talkers/strategists. Building an MVP requires you to get your hands dirty.
• Stick to your requirements.
• I cannot stress this one enough.
• This is the number 1 reason every single startup I have consulted to has struggled to launch their MVP in a timely manner.

What you DON'T want to do (stop getting distracted)

• Don't keep adding more features to build into your MVP - Keep It Simple.
• Don't go and build the world (eg: your entire product - focus on the prototype).
• Don't go and hire a team of 30 developers overseas and think it will get built faster (unless you have a team of 3-4 technical project managers to manage them).
• Normally finding 1 or 2 developers in your local country will get the job done faster and better.
• Don't change priorities every day (changing priorities constantly wastes development time and reduces your team efficiencies).

Checklist to get your MVP out FAST!

Based on the startups I have been involved in, below is a check-list of all the things you should consider while planning & building your MVP and then releasing your MVP to your customers. There are approximate time estimates against some tasks to provide you a basic guideline.

Planning & Setup

• Complete a learn canvas before anything else (~3 hours)
• This planning tool helps you determine if you have a product worth investing time in.
• Pollenizer have a great example on their tools page.
• Determine the team you will need (at a minimum you will most likely require)
• 1 designer
• 1 developer (full stack)
• 1 marketer / salesperson
• Determine the roles you will need within your team (people can own multiple roles)
• Design
• Development
• Dev ops (servers/web hosting)
• Product owner
• QA/Testing
• Marketing
• User acquisition
• Project management
• Company evangelist
• "The glue" (the person that brings everything and everyone together).
• Recruit or source your team (if you haven't got them already).
• Find an office space (co-working space is a good start) (~1 day).
• Decide on a company name (that also has a domain name available) (~1 day).
• Define a long term vision/strategy for your startup - just a sentence or 2 (~2 hours).
• Print this out and put it on the wall/door, wherever everyone will see it.
• Define your MVP goal - what you want to achieve for MVP launch (~2 hours).
• Print this out and put it on the wall/door, wherever everyone will see it.
• This will keep your team constantly focused on the end goal.
• Setup google apps/email for all your team (~4 hours).
• Create a generic email (eg: team@yourcompany.com) that you will use for all your subscriptions, billing etc.. don't use personal accounts (it makes it complicated when people move on in the future).
• Setup a task tracking or project management tool (~4 hours).
• Asana is free and simple collaborative tool.
• If you are after something more enterprise, check out JIRA from Atlassian.
• Similar to above, setup a collaboration tool (Google Docs or Confluence) for documentation so you don't loose any documentation throughout your early startup stages.
• Setup an Agile Standup Board 
• Backlog, TODO, In process, In Testing, Done
• Don't build the world
• Find and define that one feature that makes you different from your competition and focus on that.
• If you're a dating site, what makes you different? Build that feature only.
• If you're a social network, what gives you an edge to acquire users? Build that feature only.
• If you're an ecommerce site, what will make your customers purchase off you instead of your competitors?
• Fake features you are not sure about or cannot build in time:
• eg: Need "identification checks' on your website?
• It may take 3-5 days to implement with an API to provide this automatically.
• For an MVP maybe it's a file upload tool and your team manually does ID checks in the beginning (this may only take a few hours to build). This doesn't scale but remember you're building a prototype.

Design & Development

• Develop your wireframes for your website/app (~2 days).
• Define your company brand and colours (~1 day).
• Develop a company logo (~1 day)
• If you don't have resources to design your own, you can use platforms like 99designs or TemplateMonster.
• Develop your designs and user experience flows for your website/app (~3 days).
• Keep the design simple and straight forward to be built by your developer.
• Have a read through Googles Material Design Guidelines.
• Research & choose your technology stack and framework (~1 day).
• Purchase your domain name.
• Purchase and setup your website/app hosting.
• AWS provides a great infrastructure to do this and allows you to scale up as required with little effort (provided you architect you're application correctly).
• Purchase and setup your DNS hosting
• Look into using 'Route  53' on AWS.
• Define and scope your MVP development build list (~2 days).
• Breaking down into user stories.
• Prioritise your development tasks in order of what needs to be accomplished first
• This is a collaboration between your software engineers and your product owner.
• Break your scope into user stories and tasks.
• Setup a private GitHub or private BitBucket account for version control.
• If your developer doesn't know what version control is, find a new developer.
• Build your MVP (~25 days).
• Break this into 2 parts.
• Part 1 = Build the foundation that can be deployed to your test servers and be used as a feedback/review channel for the rest of your team (~15 days).
• Part 2 = Iterate and build on top of this foundation (from feedback from team) to complete your MVP (~10 days).
• Ensure your development, testing & production environments are setup (~3 days).
• Ensure they are all identical.
• Setup automated deployment (continuous delivery).
• Some tools you can look at include:
• Bamboo
• Jenkins
• Travis
• Setup Google Analytics or another analytics platform
• Do not launch without this.

Marketing & Acquisition

• Prepare your marketing plan from day one (~2 days)!
• Define who your customer is (~1 day).
• If you don't know who your customer is, figure it out before you building your MVP.
• Define your customer acquisition strategy (~2 days)
• How are you going to build your database of customers.
• Customers are very unlikely to just 'come' to your website - you need a strategy.
• Talk to everyone about your startup while you are building it.
• Don't go stealth, don't lock yourself away and think everyone will love it when you launch
• THIS doesn't happen!
• Go to meetups, meet people & get feedback from early adopters.
• Allocate your time into sections.
• A rough allocation could be:
• 10% general business setup.
• 10% wireframes/designs/logos.
• 5% content creation.
• 40% software development.
• 10% QA & testing.
• 25% marketing / reporting.
• Setup your social media accounts (~1 day).
• Facebook
• Twitter
• LinkedIn
• Instagram (optional)
• Setup a landing page to gather signups & social media followers (~2 day).
• Checkout LaunchRock to setup a free page.
• Setup a MailChimp account as a base platform for your subscribers (~4 hours).
• MailChimp is super easy to use and it's easy to export your subscribers at a later date if you want to migrate to a different platform.
• Create/practice your elevator pitch (~1 day and then ongoing).
• Know how to pitch your startup to everyone you meet (but not in an annoying way).
• Develop marketing strategy plan for launch (~2 days).
• What newsletters are you going to send?
• What SEM will you implement?
• What is your marketing budget?
• What is your social activity for launch?
• What giveaways/competitions will you run?
• What press publications are going out for launch?
• Setup a customer support tool to install on your website/app.
• ZenDesk is a great, really cheap tool.
• Select 10 potential customers for your product that you can use for early feedback (~1 day).
• Get out there and talk to your customers about your MVP and get feedback to determine if it is worth investing more time and money into your startup idea.

Testing

• Build a test plan for your MVP (~3 days).
• This can be a manual test plan in the beginning if you don't have time to build and setup automated testing.
• Test your MVP on testing servers when it is at a stable stage (~2 days).
• Do a spell check on your website before launch (~1 day).
• Release your MVP to production (and do a soft launch) (~2 days).
• A soft launch is a launch that you invite friends and family to before you release to the public (usually 2-3 days later).

Culture

• Work hard, if you are serious about your startup, put in a solid 14-16 hours a day in the beginning to get your MVP up and running.
• Have breaks or dedicate a day off to not think about your startup.
• Go out to team lunches/drinks at least once a week.

Remember...

Your MVP does not need to be perfect, this is version 1 - "a prototype". This is to test if your customer wants your product so you can justify continuing your startup and not wasting away your life on a product that no one wants.

Suggestions / Feedback?

Have something I have missed that you do when building a startup? Make a comment below and I'll add it to the post :-)

Read More