There is a persistent problem with the get_map() function now that the google api is required. Even if you ask for source=”stamen” you still get an error asking for the google api. A fix is to do the following:
Ubuntu server is a nice platform for server-related activities. Here is a short tutorial of how I updated my most current version to the latest available by rstudio.org. Here is how I got it going.
If this is your first install, you need to grab the gdebi stuff
sudo apt-get install gdebi-core
Next download the latest deb from rstudio. I typically like to try out the preview release, often stable enough to get what you want done while at the same time highlighting the latest features. When writing, it was the 1.2.1321 version.
This went out and grabbed some other libraries and installed everything for me then turned it back on. Since I had it already installed, that was the end of it. If this is the first time you are installing it, you can configure it following the installation guide here.
Today, I’ve been invited to give a talk at the University of Virginia Center for Public Health Genomics. I’ll be introducing the Population Graph framework we’ve been developing over the last decade with highlights on how we are applying it to SNP-level genomic data analysis in non-model systems.
So as a way to expand some of the analytical tools we offer the students at my work, I’m developing a version of my Data Literacy course that will use Python as well as R. There is a lot of overlap in these two languages and both are of interest to our students as they develop their toolkits. This document walks through how to set up Pweave on your machine so you can engage in a little Literate Programming (trust me, it will make your life suck a lot less. To see how to set up Atom, see my previous post.
I believe that the time is ripe for significantly better documentation of programs, and that we can best achieve this by considering programs to be works of literature. Hence, my title Literate Programming.
Knuth – 1992
If you think of it a bit, as data scientists, the documents and manuscripts we work on every day are just extensions of programs and scripts we use to do our work. However, in academia we are taught the process in entirely the wrong sequence. Traditionally, we are taught the following sequence.
We’ve are funneled by the primary interface for writing scientific documents–the word processor–into that monstrous chunk of software we use to crafted our tales about the data we were presenting. How many times have we started working on a new project and the first thing we do is fire up a editor and start an outline of a manuscript? We never really liked it but this was the main tool we were taught to use (and the crappy reference managers tacked onto them).
In a separate interface, we would perform our analyses. In my career, I’ve used:
That VAX machine over in the Math Department at UMSL. It ran SAS and I did most of my work in IML.
One-off software packages that worked on our ‘special’ kind of data we are working with. These were typically FORTRAN code written by some wizard at a far-off university. Anyone remember BioSYS from Swofford & Selander?
Workarounds in C (my own popgraph software is written in C).
Extensions that could be shoved into Excel (GenAlEx is a good example of how far you can push VB).
Scripting languages such as R, Perl (no one uses this one any longer, which is probably a good thing), Python, Julia, etc.
Then we would export the raw output to some kind of plotting software to make your graphics. I always hated this step, because inevitably, we’d have to come back and redo the graphics (higher DPI says the publisher) and we’d have to remember how we made it that last time as most of these interfaces are stupid point-and-click software packages.
The main problem is that any iteration of the manuscript would require manually going through the process or changing the text document, rerunning the analysis, then replotting the figures. Move this section up her and then go back through and make sure all your figure and table references are recovered.
But this is entirely upside-down! Instead of Communicate -> Analysis -> Visualize, our workflow should be more like:
We should be data-focused, not manuscript focused!
The research manuscript is simply an advertisement of our research and the data, it IS NOT the research or data.
Dyer – Just now!
PWeave is like SWeave (and its better version Knitr) on R. It is a tool that we can use to interdigitate our analysis and how we go about presenting it all in one place. This allows us to have a single document where we can have the data, the analyses, the output, and the verbiage that we use to describe what we are doing. This tight coupling of the data to the rest of the components helps in Reproducible Research.
To install Pweave, you need to have atom and python already configured. Then in Atom, install the following packages
Next, you can prepare a short script. Here is a fragment of one.
What this does is mix in markdown text and code. If you have not used Markdown before, it is pretty straight forward. Here are some simple rules.
A line with one or more # marks are headings.
A word or bit of text between asterisks (e.g., *this*) are italicized.
A word or bit of text between pairs of asterisks (e.g., **this**) are bold.
Links are placed in parentheses with the option to have specific word to be the link. [link](http://foo.bar)
Lists are done physically, new line with dash for unordered or new line with number as numeric.
All the python code must be within the bounds marked by the three backslashes. The code will be evaluated, from the top of the document to the bottom. You do not have to show the code for it to run.
To weave the document into HTML (we can do other formats as well but this gets us going, open the terminal and type:
And it should produce a document in the same folder but as an *.html file.
Which is pretty cool. Now, there are a lot more things you can do with markdown.
This is such a common thing to do these days, it is
easier to just post this here rather than search through my class notes each time someone asks me how to do this.
Here is the issue. Say you have some data associated with your research project and are adding to it and doing analyses. Chances are, you have it shoved into an Excel spreadsheet that is on your laptop, your home computer, the computer in the lab, a backup disk (you are keeping backups, right?), and even perhaps shared on a Cloud Drive with your collaborators/advisors/partner/whatever. Great! Now you have absolutely no way to know which version of the dataset is the real one and which are wrong.
Publishing Spreadsheets from Google Drive
In R, we can use the ability to serve out spreadsheet-like data as *.csv files using Google Drive. This way, the data are in one (and only one) location and can be accessed by anyone you would like to grant access. Here is how to set it up.
First, on Google Drive, you need to tell it to make a spreadsheet available and how to publish it. This is done from the menu as File -> Publish to the Web… A dialog box will pop up, like the one below, and let you select which sheet is published and what it is published as. The salient part here is that you should select Comma separated values (*.csv) as the output type. The URL that is provided in the image below should be copied as we will be using it in R to grab the data.
Next, you can fire up R (I use RStudio as a sane interface) and make sure you have the RCurl library installed. If not, install it like this:
So to load the file from Google Drive, we need to format the URL from Google Drive
link <- "https://docs.google.com/spreadsheets/d/1QL9fYeKkDKphba12WLVTBJrv_d1WHTc9SrZoBeIFgj8/pub?gid=0&single=true&output=csv"
url <- getURL( link )
Then open an internet connection asking for a text-based communication between Google Drive and your R session
con <- textConnection( url )
and then pull the data into R as if it was on the local filesystem.
data <- read.csv( con )
And your data should be there.
# Population SampleID X.Coordinate Y.Coordinate Cf.G8
# Min. :2.000 Min. :203.0 Min. : 346 Min. : 254 Min. :147.0
# 1st Qu.:3.000 1st Qu.:315.5 1st Qu.:1482 1st Qu.:2231 1st Qu.:155.0
# Median :4.000 Median :428.0 Median :1656 Median :2928 Median :157.0
# Mean :3.809 Mean :428.0 Mean :1747 Mean :2588 Mean :160.3
# 3rd Qu.:5.000 3rd Qu.:540.5 3rd Qu.:1914 3rd Qu.:3082 3rd Qu.:165.0
# Max. :6.000 Max. :653.0 Max. :3778 Max. :6148 Max. :199.0
# NAs :9
# X Cf.H18 X.1 Cf.N5 X.2
# Min. :149 Min. : 83.0 Min. : 83.0 Min. :148.0 Min. :150
# 1st Qu.:161 1st Qu.: 99.0 1st Qu.:107.0 1st Qu.:165.0 1st Qu.:170
# Median :167 Median :105.0 Median :115.0 Median :170.0 Median :170
# Mean :172 Mean :104.5 Mean :112.8 Mean :167.7 Mean :170
# 3rd Qu.:181 3rd Qu.:111.0 3rd Qu.:119.0 3rd Qu.:170.0 3rd Qu.:170
# Max. :519 Max. :123.0 Max. :123.0 Max. :172.0 Max. :172
# NAs :9 NAs :1 NAs :1 NAs :36 NAs :36
# Cf.N10 X.3 Cf.O5 X.4
Min. :171.0 Min. :175.0 Min. :176.0 Min. :176.0
# 1st Qu.:187.0 1st Qu.:193.0 1st Qu.:178.0 1st Qu.:182.0
Median :189.0 Median :197.0 Median :182.0 Median :194.0
# Mean :189.4 Mean :196.3 Mean :182.5 Mean :190.3
3rd Qu.:193.0 3rd Qu.:201.0 3rd Qu.:182.0 3rd Qu.:196.0
# Max. :205.0 Max. :205.0 Max. :202.0 Max. :204.0
NAs :13 NAs :13 NAs :8 NAs :8
This past summer has seen some rather spectacular cases of where people have run afoul of dangerous flora, the most recent of which was a college student after a runin with Giant Hogweed–the results were not good.
Being new to the Windows platform, I’m on the look for a good text editor that can do the myriad of tasks that we do each day. Notepad is not an option, let’s be real. I’m looking for something that can be extended and has been designed from the bottom up for wrangling text and writing code. Ultimately, I would like something that is amenable to teaching both R and Python using a single interface. RStudio is great for R but sucks for Python. Juypter notebooks are clunky and toy-like.
Atom is created by the Github folks and is integrated into ‘the mothership’ repository. Here is what I did to get it up and running and having Python running correctly.
Packages are extensions to the main editor that accomplish some function to make you life a bit easier. Here are some of the ones I find helpful. You can find packages and install them using Settings -> Install. Then search for the packages and hit the install button.
If you have scripts and/or code that is longer than a single page (and who doesn’t) minimap provides a graphical depiction of your code on the right-hand side of the window to allow you to easily jump up and down the file. Here is an example on an R script.
Script is a package that runs code in the editor directly. This means you can run individual lines as you develop and look at the output. Very helpful.
There are a ton of themes, both overall for the editor as well as syntax highlighting, available. To install one, select Install -> Theme and type a name from Settings. Here is the atom-material-syntax being installed.
Once installed, you can change both the UI and the syntax colors.
As I expand more into using Atom, I’ll add additional posts showing how I have configured it for use in my daily coding activities.
Today, we had the first event in the Global Environment Speaker Series hosted by the VCU Center for Environmental Studies and the Departments of Geography & the Environment and Environmental Studies from the University of Richmond—a set of units that already has a lot of collaborations among faculty, curricula, and students.
Syndication is a process whereby you can post something to your site and other locations will detect that you have posted something and then pull in the content to their site, making it look like you wrote it and posted it on their site. Is that clear? Here is my use-case:
I post everything I do to rodneydyer.com.
For things that I want to be shown on my work page (https://dyerlab.org), I select a particular Category or Tag for the post.
My work site monitors and any time something at rodneydyer.com comes up with the key Category and/or Tag, dyerlab.org pulls the content of the post in and formats it to look just like I wrote it for that site.
This is particularly interesting for teaching and other uses. If a class uses WordPress for its webpage, students can provide content for that class page by publishing on their own site. This allows each student to create a "Digital Portfolio" of work that they maintain (see my thing on Content Silos for more on this).
I'm going to use the FeedWordPress Plugin for this because it was the one that my university uses and I want to standardize the approaches.
To install it, go to Plugins->Add New and search for it. Install & Activate .I'm going to use a new Category, named Dyerlab, to trigger the syndication. So I add a new one.
OK, now on my personal page, I have a category "Dyerlab" that I will attach to things that I want to show up on my Dyerlab WordPress site. To make the connection, we need to get the category feed address. Unless you are changing something drastic it has the following structure:
which in my case is:
You can try it out and you should see (if you have any posts with that category published) a list of just those posts. If so, perfect. If not, then you either have not posted anything with that category or you have not set up the category correctly. Go back and check.
Now, I need to set up the other site, in this case my laboratory site, to monitor my personal site, and any time something is posted, grab it. Go open your other site and make sure the plugin is installed. This site will "Pull" the posts from the original site. Click on Syndication in the bottom left panel and you will open the settings page.
In the "New Source" box, paste in the category address from your other site. In my case I pasted in
You will be taken to a verification screen where you can verify that things are working properly and select the correct feed type. There is a 'verify' link that you can use to make sure it is providing good input. After you select which kind of feed you want, you will be redirected back to the list, as above, but with your new feed in it.
Now, when I write something (like this post) on my site, it will automagically show up on my laboratory site as well. The Cool thing is that wherever it is displayed, it is reformatted to look as if it belonged at that location. Here is this post on my personal site.
and on my laboratory site
are identical in content, though are individually styles. Pretty cool!