Fetching stock data using a parameterised Execute R operator

I'm currently delivering data science lectures at the University of Chichester and RapidMiner is part of what I use to teach. And very good it is too. I recently found myself helping my students to get some up to date stock market data. Rather than manually downloading this, I thought I would use RapidMiner with the tidyquant R package and do it automatically. The Finance and Economics Extension seems to be out of date so isn't an option.

My idea was to define a list of stock symbols such as "AAPL", "BTC-USD" and so on and run the Execute R operator in a loop with each symbol individually.

It turns out there isn't a way to parameterise the Execute R operator so I had to invent one.

Basically, I use the Loop Parameters operator to set multiple values for a macro located inside it. This macro is used to create a one row example set with the value of the macro. This example set is then passed to the Execute R operator where the R script uses it as a parameter to drive the rest of the script. It's clunky but it works.

This approach could be adapted to allow R scripts to be run as part of a more complex modelling process. Relatively tough to do but feasible.

Here's a link to the process.

You'll need the R Scripting extension and you will also need to ensure that R is running on your machine with the data.table, tidyverse and tidyquant R packages all installed.

If RapidMiner enhances the Execute R operator to take parameters, (which would be a good enhancement), then this work around will not be needed anymore.

RapidMiner Master Certificate and other news

I'm pleased to announce that I've successfully passed the RapidMiner Master Certificate examination.

It took a fair amount of revising and I'm very pleased with the outcome.

I will now be able to devote more time to the next set of RapidMinerResources videos of which these are currently under construction...

• Installing RapidMiner Server
• Loop Parameters Grid
• Pivot
• De-Pivot
• Loop Collection
• Nested Loops

Check back often to get more information about these and what others I am planning.

The Write Special Format operator

Here is a complicated process that does quite a lot...

Deep breath, here we go...

1. Loop through all the process files in a repository (make sure you point it to your location) and read each one in as a document.
2. Convert the documents to normal examples within a single example set.
3. Create an attribute called `description` that contains the text within the top-level comment for each process. This uses Ninja XPath (actually it doesn't but I wanted to use the word Ninja).
4. Do gymnastics to reformat the contents of the attribute. This uses Ninja regular expressions (actually it doesn't but the usual rule is all regular expressions require Ninja like skills). Newlines and linefeeds are included - this is perhaps the interesting part.
5. Rename to make the name of the attribute easier to understand.
6. Select only the attributes of interest.
7. Filter out all where there is no description.
8. Write the example set to a file (make sure you set this to somewhere you want to write files). This uses the `Write Special Format` operator which was the only way I could find to get this to work.

Why?

The comment view can contain html. This allows basic formatting and structure to be defined so that when a process is opened, the text in the comment view is displayed with this formatting and structure applied. The process above allows the html tags to be extracted so they can be re-used if you want to use the formatting and structure somewhere else like a document or index. Time is precious, anything that allows re-use and avoids typing is good.

The `Write Special Format` operator is especially nice because it allows precise control to be exerted over what is to be written. Dare I say Ninja like control?