Random walk in 3D

Here's a process that draws a pretty picture of a random walk in 3 dimensions in 3D.

OK - you have to work at it. But by squinting at the screen and going cross-eyed so the right side image appears in your left eye and the other in your right, you should see a 3 dimensional view of a random walk.

You may have to adjust the width of your window and who knows what else to make the images appear side by side. Persistence is valuable.

The colour of the points is set by the id. Blue is near 0 and red is at the end of the series. In this case, 2000 data points.

The process uses macros and the Loop Example operator to calculate a random amount to add to each data point. Use of the Integrate operator builds a cumulative example set where each example depends on the ones before it.

The process has the random seed set to -1. This means that it is very unlikely that the picture shown here will ever be recreated again.

Complex numbers

I stumbled upon an unadvertised feature in the generate attributes and generate macro operators in RapidMiner Studio 6. It's possible to use the square root of -1, denoted by the reserved symbol i, in the function expressions when creating attributes and macros. There are also some new functions:

• re()
• im()
• conj()

Use of a complex number to generate an attribute results in the real part being used only whereas generating a complex macro like this

CN = -3+2*i

results in a macro value that looks like this:

(-3.0, 2.0)

This would be -3+2i.

Can a macro, once created, be used as an argument to create other macros or attributes?

The answer is no but there is some fiddling around that can make some things work.

If my macro is generated like this

CN = -3+2*i

I can create another macro like this

CN_1 = replace("%{CN}", ", ", "+i*")

I then get this value for CN_1

(-3+i*2)

I can then put this result in an attribute as a string. I can also use functions like re() and abs() directly on this so for example I can create an attribute like this (note that I am passing a string to the abs function that happens to be a valid complex number)

CNLen = abs(%{CN_1})

Sure enough, the attribute contains the expected value of 3.606.

I'm not sure if this is a taste of things to come or whether it's unintended. I suppose it would be cool to have the ability to manipulate complex numbers.

Bulk export of processes

I am doing something at the moment which requires me to to export a load of processes contained in folders within a single repository to a single disk location. Rather than do it one by one which is error prone and time consuming, I decided to make a RapidMiner process to do the export.

It turns out that the files in the repository with the extension .rmp are valid xml files that can be imported so all I did was point a Loop Files operator at the folder where my repository was and looked for files ending in .rmp. Inside this loop, I used the Generate Macro operator to generate the new location and the Copy File operator to copy the file from the repository to the new location.

The location of the repository and the location where the files to be copied are defined in macros in the process context. Set these to the values you want. Note that on Windows machines it is necessary to use double backslashes to delimit folders.

The process is here.

Parsing macros with values containing backslashes

Let's imagine we are looping through some files and we happen to know that the files are buried somewhere in folders called chapterNN where NN is a number. How can we extract information about the location for each file individually?

Within a Loop Files operator, the parent_path macro would take values like this within the loop (I'm assuming Windows obviously).

c:\myGiantFolderStructure\mybook\chapter01
c:\myGiantFolderStructure\mybook\chapter02
...
c:\myGiantFolderStructure\mybook\chapter11

If we wanted to extract the last part of the folder name including the number to use in the loop we could try the Generate Macro operator with the replaceAll function.

The basic idea would be to configure the operator like so.

chapter = replaceAll("%{parent_path}", ".*(chapter.*)", "$1")

This matches the entire macro with a regular expression but locates the word "chapter" and whatever follows it inside a capturing group. This replaces the entire value of the parent_path macro. The result should be a new macro called chapter with values from chapter01 upwards.

Unfortunately, this doesn't work because the backslashes cause trouble. I'm guessing but I think the replaceAll function (and its siblings, replace and concat) try to parse the string within the macro and get confused by treating the backslash as an escape character.

Fortunately, there is a solution: a little known function called macro. This function simply returns the string representation of a named macro.

The expression would then look like this.

chapter = replaceAll(macro("parent_path"), ".*(chapter.*)", "$1")

Knowing that backslashes are processed enables to us work out how to pass them simply by escaping them. If we felt we needed a more sophisticated match inside the capturing group to ensure we picked up numbers, we would do the following.

chapter = replaceAll(macro("parent_path"), ".*(chapter\\d+)", "$1")

This matches only if there is at least one number after the word "chapter". The double backslash becomes a single backslash when the regular expression is evaluated and \d+ means one or more numbers.

Deleting attributes with 2 valid values and the rest missing

For some reason the other day, I can't remember why, I had to delete attributes with two valid values and all the rest missing. It followed on from this process.

So I made this process that finds all attributes with a specific number of valid values and removes them (the attributes).

This example uses sample data so deletes attributes with 7 valid values but I'm sure you'll get the idea.

Generate Macro bonus features

I was pleased to discover that some of the functions available in the operator "Generate Attributes" are also available in the "Generate Macro" operator.

For example the following functions work.

concat
contains
matches
index
str
upper
lower
escape_html
replace

and I imagine that similar text processing functions will also work.

I tried date_now() and some other date functions and I got a result that looked like an error (actually quite an interesting error). So I assume date functions cannot be used.