thewheeldogg.com

Getting started with R (Rstudio, Rshiny) running in a docker container.


| |

Rstudio setup in Docker.

I’m was having some difficulty getting Rstudio running on windows through the Windows subsystem for linux (WSL). I’d like to build using Docker as then I’ll be able to develop in R in a docker image that I can move between windows and ubuntu with no problems.

docker

Full documentation for this available from rstudio

rocker and rstudio.

https://hub.docker.com/r/rocker/rstudio/ 1st command to start the container as this will ensure we have the correct R setup.

docker run --rm -ti rocker/r-base

rstudio-server will probably run on ubuntu 18.04.

Try uninstalling and reinstalling it ?

create your docker image of

sudo docker run -d -p 8787:8787 -e PASSWORD=<password> --name rstudio rocker/rstudio

might take some-time the first time you download it.

check the running docker container.

docker container ls and stop it ` docker container stop rstudio and start it docker start rstudio`

Additional

important for tidying up your docker images. https://www.digitalocean.com/community/tutorials/how-to-remove-docker-images-containers-and-volumes

Todo.

Check out using more distros on windows. windows docker install Windows and WSL working flawlessly

Building my own dockerfile from the base image from Rocker with additional packages. https://datawookie.netlify.com/blog/2017/07/rstudio-environment-on-digitalocean-with-docker/

Rstudio and RShiny running together in Docker.

Inspiration from here https://bioconductor.org/packages/release/bioc/vignettes/sevenbridges/inst/doc/rstudio.html docker pull sevenbridges/sevenbridges-r

enter the container. docker exec -it <container-id> bash

Mounting your local directory in the docker container. docker run -v /c/Users/Shane.Whelan/:/my_mnt -d -p 8787:8787 -p 3838:3838 –name rstudio_shiny_server sevenbridges/sevenbridges-r

fixing up the django conda environment.

  • install bzip2 for installing anaconda. https://www.howtoinstall.co/en/debian/stretch/bzip2
  • download conda run bash installer.

Reticulate package! Ability to read python objects/functions in R.


| |

drawing

Documentation for reticulate package.

official https://rstudio.github.io/reticulate/index.html

nice run-through https://cran.r-project.org/web/packages/reticulate/vignettes/package.html

Overview.

As a Python and R programmer it is a great asset to be able to transfer and utilize both these tools for data analytics. A great article for learning more about the differences between Python and R, and the utility for using both is outlined on Rbloggers.

Rbloggers post on R and python

Firstly there are instance where we want to mess around with data in R make use of the tidyverse paradigm and save this data that it’s available than for python. An example of this is when we have wrote our manipulations/statistics in R and prefer to use Python for interacting with the web and APIS. I will write about this in a future blog how I implemented R and Python to extract transform and load directly to Tableau server.

In this blog post however I want to talk about something that I feel will be really useful in terms of being the inverse of what is mentioned above. In this case we want to be able to take data/objects/classes/modules from our python environments and make this data available in R for data manipulation and visualization.

I started messing around with the reticulate package and feel this is an excellent solution to accomplish this task.

Methods.

Code available on Githbub

  • The Python file shows how we can access our model from django unleashed. The django project in this case is suorganizer.

  • The R script than points to a directory of this R script and is able to load the objects within.

Next steps.

Here we only have a very basic example working. The idea of this work will be to build an Rshiny app that we can intergrate in Django utilizing the django models and queryset objects. This means that primarily our manipulations and data extraction layer will be done in python, while visualizations and statistical analysis can be done in R. I feel this will be a great way to utilize the power of both tools in analytics.

Hosting RShiny apps within Django framework.


| |
drawing drawing drawing

Section 1. Introduction.

Excited to learn Django as a web framework.

I really would like to include some of the prowess of shiny applications and implement them in the Django framework.

Main draws for using Django for hosting Shiny applications is that we have a central web framework for all of the applications and we can authorize differen user privileges on login.

Excellent resources.

  • https://djangobook.com/tutorials/django-overview/
  • Django unleashed.

Objective.

  • Build a webapp to host a simple shiny application with authentication.
  • Host our Django application on Heroku.

Section 2. Methods.

  • Getting started : Conda environment setup django and run server.
  • Start building and playing with Django.
  • Setup Django to show Shiny app.
  • Deployment and docker, also deployed on Heroku.

2.1. Getting started. Conda environment setup django and run server.

For new development work we can use Conda to manage out python environment. Environment. Source activate py3 (py3) bash-3.2$ conda install django

Django docs verify django install.

import django print(django.get_version()) 2.1.1 Next we want to run a simple application using django

2.2. Start building and playing with Django.

At the moment I’m learning all about Django and using Django unleashed as my bible for this project. I’m finding this book an excellent resource so far. Please refer to blog posts on Django on this blog.

2.3. Deploying Django app on Heruko

Check out this blog which is an excellent resource for what we are trying to achive. http://pawamoy.github.io/2018/03/15/django-auth-server-for-shiny/#overview

Code related to this blog post available here on github.

I then forked the repository and made some changes to the shiny docker file. This is available on my github. This contains the code and instructions as provided to run the docker images and get the Django app up and running.

Visiting http://localhost:8000/, will look like below. drawing

I included the code for a simple example shiny application in the file shinyapp/app.R

drawing

This is great because we can now see that we have to login using our credentials that we created when we ran “sudo make all” to visualize and use the application.

Next step is to deploy this on the web itself, and for this we can use Heroku.

2.4. Deploying containerized Django app on AWS.

The docker application in the previous step is containerized. The next step is to deploy this docker app with multiple docker images on Heroku.

You can do this simply by having an EC2 instance on AWS, downloading the

When you run “make up” after “sudo make all” for creating the docker images you should get the following output. Browse to the correct

ubuntu@ip-172-31-5-114:~/docker-django-auth-shiny$ make up
Starting docker-django-auth-shiny_database_1  ... done
Starting docker-django-auth-shiny_shinyapp_1  ... done
Starting docker-django-auth-shiny_memcached_1 ... done
Starting docker-django-auth-shiny_djangoapp_1 ... done
Starting docker-django-auth-shiny_nginx_1     ... done
Attaching to docker-django-auth-shiny_memcached_1, docker-django-auth-shiny_database_1, docker-django-auth-shiny_shinyapp_1, docker-django-auth-shiny_djangoapp_1, docker-django-auth-shiny_nginx_1
database_1   | 2018-10-05 12:54:02.138 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
database_1   | 2018-10-05 12:54:02.138 UTC [1] LOG:  listening on IPv6 address "::", port 5432
database_1   | 2018-10-05 12:54:02.146 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
database_1   | 2018-10-05 12:54:02.172 UTC [21] LOG:  database system was shut down at 2018-10-05 12:45:08 UTC
database_1   | 2018-10-05 12:54:02.179 UTC [1] LOG:  database system is ready to accept connections
shinyapp_1   |
shinyapp_1   | R version 3.4.4 (2018-03-15) -- "Someone to Lean On"
shinyapp_1   | Copyright (C) 2018 The R Foundation for Statistical Computing
shinyapp_1   | Platform: x86_64-pc-linux-gnu (64-bit)
shinyapp_1   |
shinyapp_1   | R is free software and comes with ABSOLUTELY NO WARRANTY.
shinyapp_1   | You are welcome to redistribute it under certain conditions.
shinyapp_1   | Type 'license()' or 'licence()' for distribution details.
shinyapp_1   |
shinyapp_1   |   Natural language support but running in an English locale
shinyapp_1   |
shinyapp_1   | R is a collaborative project with many contributors.
shinyapp_1   | Type 'contributors()' for more information and
shinyapp_1   | 'citation()' on how to cite R or R packages in publications.
shinyapp_1   |
shinyapp_1   | Type 'demo()' for some demos, 'help()' for on-line help, or
shinyapp_1   | 'help.start()' for an HTML browser interface to help.
shinyapp_1   | Type 'q()' to quit R.
shinyapp_1   |
shinyapp_1   | > shiny::runApp('/root/shinyapp', port=8100, host='0.0.0.0')
shinyapp_1   | Loading required package: shiny
shinyapp_1   |
shinyapp_1   | Listening on http://0.0.0.0:8100
djangoapp_1  | [2018-10-05 12:54:02 +0000] [1] [INFO] Starting gunicorn 19.7.1
djangoapp_1  | [2018-10-05 12:54:02 +0000] [1] [INFO] Listening at: http://0.0.0.0:8000 (1)
djangoapp_1  | [2018-10-05 12:54:02 +0000] [1] [INFO] Using worker: sync
djangoapp_1  | [2018-10-05 12:54:02 +0000] [7] [INFO] Booting worker with pid: 7
djangoapp_1  | [2018-10-05 12:54:02 +0000] [8] [INFO] Booting worker with pid: 8

This will look like this. drawing

Please note: In order to install and run shiny applications I believe you require more than 1gb of Ram. Thus, it’s neccessary to ensure your instance state on AWS is over this threshold, otherwise it was taking a pretty long time create the docker images.

That’s the great thing about AWS that you can ramp up the RAM and disk space how you see fit !!!

2.5. Alternative deploying containerized Django app on Heroku.

Note. I might revisit this to deploy on Heroku directly as it should be possible to deploy directly here drawing

Following this to push a multiple images dockerfile to Heroku. https://devcenter.heroku.com/articles/container-registry-and-runtime

Should be relatively straightforward as per below. https://devcenter.heroku.com/articles/django-app-configuration

Note. I was getting 404 issues when getting to release the deployed images but I will look at this again. For not it is good news that we can host these on AWS in the meantime.

Section 3. Alternative way to use the Django CMS

Django and Rshiny together http://pawamoy.github.io/2018/03/15/django-auth-server-for-shiny/#overview

3.1 Django CMS

Video on CMS in Django here https://github.com/mfcovington/djangocms-shiny-app http://pawamoy.github.io/2018/03/15/django-auth-server-for-shiny/

3.2 Install Django CMS.

http://docs.django-cms.org/en/latest/introduction/install.html

source activate py3
pip install --upgrade pip
pip install djangocms-installer

3.3 Create project.

cd ... (allows you to use the djangocms command)
mkdir djangoshinycms-project
cd djangoshinycms-project
djangocms -f -p . mysite (alternatively should I follow djangocms -p . $PROJECT_NAME ???)

3.4 Start up and run the server.

python manage.py runserver

Now you will see this content management system on local host on drawing

3.5 Add CMS_shiny to settings.

Go to settings and unders installed apps add CMS_shiny. Than run

pip install djangocms-shiny-app
manage.py check.
E. Start up and run the server.

edit the djangocms to be able to attach shiny applications.

mkdir example of iframe below.

3.6 Alternatively including Shiny applications in iframes.

################ If you want to skip and go straight to running the docker image. Creating the docker image. Clone the directory and build first

sudo make all .

This will probably take about 5 minutes to run and create all dependencies.

Then run the docker image.

sudo make up .

################ Alternatively test running of the shiny applications themselves. sudo R -e “shiny::runApp(appDir=’shinyapp’, port=8100)” Test running of the Django app. python djangoapp/manage.py runserver localhost:8000 ################

Section 4. Results

  • We now have a solution to build shiny applications and a foundation of creating nginx credentials for creating different users with access permissions.
  • This was a good demonstration of getting up and running docker images that we can create these locally and run on AWS.
  • Shiny and Django will be an awesome combination for producing interactive visualization applications.

Section 5. Discussion

  • We build and deployed an AWS service using the django framework with an Rshiny app embedded and credentials via nginx.
  • Next steps will involve continuing with Django unleashed and CMS features and seeing how to house and build out multiple Rshiny applications.

Appendices and additional resources.

Docker. https://docs.docker.com/compose/django/#define-the-project-components

Heroku ! Deploying a webapp representing a slides presentation.


| |

Django Unleashed ! Part 1: Core features.


| |

drawing

Using this page to keep track of progress and notes while reading the Django Unlimited book. Pinkham, Andrew. Django Unleashed (p. 3). Pearson Education. Kindle Edition.

I’ve kept some of my key notes from this book in this page.

The book is dedicated entirely to back-end programming and generating dynamic webpages with Django.

Chapter 1. Starting a New Django Project: Building a Startup Categorizer with Blog

Ref Note. As a developer using a framework, you are simply adding to or directing the behavior provided by other developers.

Our immediate interest with django-admin is the startproject subcommand, which automatically generates correct project scaffolding with many, but not all, of the expected Django conventions.

Run this command to start your project.

$ django-admin startproject suorganizer

Create a database to run django.

$ ./manage.py migrate

Run server.

./manage.py runserver

Success. Visit http://127.0.0.1:8000/ and you will see Django installed and running fine !

Next within your django project you can start an app. Manage.py creates the app, whereas django-admin is used to create the project. A project is made up of more than one apps.

create apps

$ ./manage.py startapp blog
$ ./manage.py startapp organizer

Next you can use suorganizer directory and the settings file to tell the project about the created applications.

Chapter 1. Review and takeaways

  • Concept of “inversion of control”
  • django-admin for creating the project scaffolding
  • manage.py tool to create apps and run the test server

Chapter 2. Hello world. Building a basic webpage in Django

Create new app

$ ./manage.py startapp helloworld

Next we create the greeting() function. We then define a URL in the suorganizer urls.py

Documentation. Function views 1. Add an import: from my_app import views 2. Add a URL to urlpatterns: path(‘’, views.home, name=’home’)

Thus, as we set the URL to empty. When we run

$ ./manage.py runserver.

we are directed straight to our helloworld app ! Great, this webpage just uses the controller functionality of the MVC architecture.

Chapter 2. Review and takeaways

  • A webpage is a python callable called a view (i.e. from helloworld.views import greeting) is added to /suorganizer/urls.py.
  • Views and URL represent the controller part of Django.
  • Again inversion of control we directed django to our functions and it handled everything else.

Chapter 3. Programming Django Models and Creating a SQLite Database

This chapter is all about database modelling in django. Database rows in Django are called model instances.

In an object oriented python program, you can restrict access to methods and variables. This can prevent the data from being modified by accident and is known as encapsulation.

We start by creating a Post model for our blog post data. To create a model, we can simply create a Python class that inherits from models.Model, as shown below.

from django.db import models


# Model Field Reference
# https://docs.djangoproject.com/en/1.8/ref/models/fields/


class Post(models.Model):
    title = models.CharField(max_length=63)
    slug = models.SlugField()
    text = models.TextField()
    pub_date = models.DateField()

With our Post model firmly defined in our blog app, we can now write the models necessary to run the startup organizer section. We define these fields for our databases in the classes within organizer/models.py.

Next we create relationships between models that were carefully destined. These can be one to many, many to many relationships etc.

One to many relationship.

ie. add a foreign key model to our Newslink model. startup = models.ForeignKey(Startup) ie.

class NewsLink(models.Model):
    title = models.CharField(max_length=63)
    pub_date = models.DateField()
    link = models.URLField()
    startup = models.ForeignKey(Startup)

Many to many relationship.

Same as a above. There is no direction in many to many relationships. tags = models.ManyToManyField(Tag)

Next part is about defining the model behaviour.

For instance field uniuqeness (ie. ‘slugs’).

Adding methods to django models.

Because Django models are Python classes, developers can add behavior to the model by writing class methods. Doing so is as straightforward as writing any Python method because fields act like class attributes within Python.

While the string representation method is quite helpful, as we shall rapidly discover, the true takeaway here is that all fields are easily manipulated because Django is python.

Controlling Model Behavior with Nested Meta Classes

Django will look for the Meta class in the Tag class and then search for known options, affecting the behavior of Tag instances or groups of Tag instances.

Using Django to Automatically Create a SQLite Database with manage.py

Now that we have the data structured we can use Django to build our database.

Note. You should always use Django to build your database. One of the biggest mistakes beginners make is to try to create the database separately from Django, either before or after creating their Django models. This is a very clear instance of developers “fighting the framework.” Not only is it an unproductive use of time, it is actually counter-productive. You will always want to define models in Django and then make Django generate the database for you.

In Django, models and the database schema are reflections of one other.

performing a well controlled migration. Migration files act almost like version control of the database.

  1. Create or change a model in Django,
    1. Generate a migration file,
    1. Use the migration file to create/alter the database.

How to perform a migration.

(py3) bash-3.2$ ./manage.py  check

(py3) bash-3.2$ ./manage.py makemigrations
Migrations for 'organizer':
  organizer/migrations/0001_initial.py
    - Create model NewsLink
    - Create model Startup
    - Create model Tag
    - Add field tags to startup
    - Add field startup to newslink
Migrations for 'blog':
  blog/migrations/0001_initial.py
    - Create model Post

(py3) bash-3.2$ ./manage.py  migrate

Alternatively run the migrations one at a time for each of the apps

The key takeaway from all of this is that models allow Django to automatically create and modify databases thanks to migrations. Django is quite intelligent about this process and focuses on shortening code. Django automatically supplies a primary key to models and sets up indexes for these keys. Furthermore, Django allows us to avoid the tedious and repetitive process of defining many-to-many relations as full tables and instead provides a field to shorten our code and the process.

Manipulating Data in the Database: Managers and QuerySets

Django Object-Relational Mapper The Django web framework includes a default object-relational mapping layer (ORM) that can be used to interact with application data from various relational databases such as SQLite, PostgreSQL and MySQL.

Get up and running with the jupyter notebook (Note. Please insure you have Jupyter installed using pip for your current dev environment). Change directory into the code repository for Django unleashed than run.

jupyter notebook

Greeted by the following. jupytert

However, in our case it is better to put the jupyter notebook into the /suorganizer folder that we are currently working on. This allows us to test directly our own build.

Note. to install pylibmc which is required for this demo on MAC. please run brew install libmemcached beforehand or the install will not work.

#####

Chapter 4. Review and takeaways

  • This chapter was all about templates, this represents the View portion of the MVC architecture.
  • Two parts: the templates themselves and the renderer.
  • In the template we were introduced to the django template language. This allows us to perform loop logic, if statements, operators and filters.
  • A variable may have data attributes accessed just as in Python via a period: .
  • Additionally, filters, which are functions dedicated to formatting or modifying variables, may be applied with a vertical bar (with no spaces in between): .
  • Some filters allow or expect a single argument, which may be passed in with a colon and enclosed in quotations if the argument is not an integer: .
  • Filters may also be chained: .
  • ** Template tags ** in the django template language.
  • The extends tag, which allows templates to inherit markup from another template.
  • Template tag conditions and loops typically create scope, meaning they must be closed.
  • For loops in templates, Both work the same way they do in Python, but they must be closed, a tag that imitates the else condition of a for loop in Python, in that it allows for the condition of an empty list passed to for.
  • The template documents may be loaded and rendered in Django through the use of the template package.
  • We can use the Django console to prototype our code and import the tamplate package and rendered. the rendered string from the context can be rendered into a view using HttpResponse.