AWS, R, RStudio, Parallel processing

In this post I will share my experiences with using a spot AMI instance for heavy parallel processing of R scripts.

Fist we start a Rstudio AMI:

Start a spot instance:

Request:

Search AMI:

Search for Rstudio:

select the needed compute power:

Its recommended to look at the pricing history:

In this case, we select a General Purpuse 16 CPU machine, which has a fair price per hour. Since we only use the machine for some heavy processing, we will decommission it in a few hours.

Leave the other settings as is.

select next, be sure to create a new key pair, in case you do not have the original key pair.

Its also good to add a new security group which has port 80 open:

open the ports:

The click 'Launch' instance, and then click op the capacity link;

Look up the public IP:

You should now be able to log on:

Log on credentials can be found here:
http://www.louisaslett.com/RStudio_AMI/

Now that the server is up and running we can start using R packages to allow parallel processing:

  
install.packages("doParallel",dependencies=TRUE)
install.packages("doMC",dependencies=TRUE)

We use the package

  
library(doParallel)
library(foreach)
library(doMC)

In order to initialize we;
- set the number of cores.
- initialize a cluster
-and we need to export our functions, variables and datasets to the clusters:

  
#prep parralel processing
# Calculate the number of cores
no_cores <- detectCores() - 1
# Initiate cluster
cl <- makeCluster(no_cores)
registerDoMC(no_cores) 
clusterExport(cl=cl, varlist=c("splitter", "create_ngram_table"))

There are a number of options for parralel processing roughly speaking you can choose from ParLapply and forEach. When you would like to split your dataset in multiple subsets and want to have a worker node perform an operation this can easily be done with a forEach, see example below. In both cases a list is returned. While we usualy would like to have a full dataset in return. In order to achieve this result I used the code do.call("rbind", result).

Many more examples can be found on: http://gforge.se/2015/02/how-to-go-parallel-in-r-basics-tips/

  
 result = foreach(j=seq(1,nl, by=(nl/parts))) %dopar% {
      }
    }
  }  
  do.call("rbind", result)
}

Text prediction

In this post I would like to show off the text prediction app I created to obtain my Data Science specialization:

Market Basket Analysis (Association Rule Learning) with Power BI (DAX) and R

Introduction

In this post I will show how to run an R script from Power BI which will execute an Association rule learning script to perform market basket analysis.

In this example we will not look at products sold, but products sharing shelf space.

The dataset

Our basic dataset looks like this.

 Our products:

The distribution / presence of products on the shelf of a customer:

The Power BI building blocks

The data model

As for the DAX part we will start with this post of Marco Russo and Alberto Ferrari.

So the data model in Power BI looks like this:

The R visualization

We will look at the DAX part later on. First we add an R component with a script that will return the AR rules it found.

The table contains the basic output that is to be expected from AR. We will try to build these measures in DAX later on.

The R script

As for the R script it looks like this:

   
save(dataset, file="C:/TFS/dataset.rda")

library(arules, lib.loc="C:/TFS/Rlib/a/" , logical.return = FALSE,
        warn.conflicts = F, quietly = T,verbose = F)
library(plotrix, lib.loc="C:/TFS/Rlib/p/" , logical.return = FALSE,
        warn.conflicts = F, quietly = T,verbose = F)

dataset = cbind(dataset, 1)
colnames(dataset) = c("ProductID", "CustomerID", "Waarde")
reports = xtabs(Waarde~CustomerID+ProductID, data=dataset)
reports[is.na(reports)] <- 0
rules <- apriori(as.matrix(as.data.frame.matrix(reports)),parameter = list(supp = 0.03, conf = 0.5, target = "rules"))
t = inspect(head(sort(rules, by ="support"),15))

par(mar = c(0,0,0,0))
plot(c(0, 0), c(0, 0))
if (is.null(t)) {
  t = data.frame("no rules found")
  text(x = 0.5, y = 0.5, paste("No Rules found"), 
       cex = 1.6, col = "black")
} else {
  addtable2plot(-1, -1, t, bty = "n", display.rownames = F, hlines = F,
                vlines = F)
}

Unfortunately Power BI initializes a new R sessions each time the R visualization is run / cross filtered.  Therefore I tried to use a much base R as possible. As for the libraries that need to be loaded. I put these in a separate folder on my local drive and specified the folder name in the library command.

Building it in DAX

Support

The output of the arules R script can be built in DAX whenever it concerns single item combinations, so X -> Y. So not A, B -> Y.  The 'support' measure is basically the '[Orders with Both Products %]' described by Russo and Ferrari. Just to show how its implemented on our dataset.

  
Customers with Both Products % = 
IF (
    NOT ( [SameProductSelection] );
    DIVIDE ( [Customers with Both Products]; [Unique Customers All] )
)

The building blocks of this formula:

Same product selection, since this is useless. SameProductSelection = IF ( HASONEVALUE ( Products[ID] ) && HASONEVALUE ( 'Filter Products'[ID] ); IF ( VALUES ( Products[ID] ) = VALUES ( 'Filter Products'[ID] ); TRUE ) )
Customers with both products: Customers with Both Products = CALCULATE ( DISTINCTCOUNT ( Distribution[Customer ID] ); CALCULATETABLE ( SUMMARIZE ( Distribution; Distribution[Customer ID] ); ALL ( Products ); USERELATIONSHIP ( Distribution[Product ID]; 'Filter Products'[ID] ) ) )
Number of customers in total: Unique Customers All = CALCULATE ( DISTINCTCOUNT ( Distribution[Customer ID] ); ALL ( Products ) )

Confidence

   
Confidence = [Customers with Both Products] / [Unique Customers LHS]

Unique Customers LHS:

   
Unique Customers LHS = DISTINCTCOUNT(Distribution[Customer ID])

Lift

Lift = [Confidence] / [Proportion Product RHS]

Proportion product RHS:

Proportion Product RHS = Distribution[Unique Customers RHS] / [Unique Customers All]

Unique customer RHS:

 
Unique Customers RHS = 
CALCULATE (
    DISTINCTCOUNT ( Distribution[Customer ID] );
    CALCULATETABLE (
        SUMMARIZE ( Distribution; Distribution[Customer ID] );
        ALL ( Products );
        USERELATIONSHIP ( Distribution[Product ID]; 'Filter Products'[ID] )
    ); ALL(Products)
)

You can download the Power BI file here.

In this video you see the Power BI file in use:

Image to data frame to ggplot heatmap

Image to DF to ggplot

Steve de Peijper

25 mei 2016

In this post we will convert an image to a dataframe to display the image with ggplot. In a later post we will compress the image.

library("png", lib.loc="C:/TFS/Rlib/")
require(ggplot2)

## Loading required package: ggplot2

This is the original image:

.

The work

plaatje = readPNG("c:/TFS/test.png")
x <- data.frame(1:100,1,1,1,1,1)
colnames(x) = c("t", "x","y", "r", "g", "b")
for (a in 1:3) {
  teller = 1
  for (i in 1:length(plaatje[,1,a])) { #Hoogte
    for (j in 1:length(plaatje[i,,a])) { #Breedte
      x[teller,c(1:3,3+a)]  = c(teller,j,i,plaatje[i,j,a])
      teller = teller + 1
    } 
  }
}


ggplot(data=x, aes(x=x, y=y, fill=rgb(r,g,b))) +
  geom_tile() +
  scale_fill_identity()