Computation

Binder Binder Binder

We will be using (I)Python as our programming language and QGIS for basic GIS exploration. In order to use some of the material available on this website and to share your material with others you should create a GitHub account for yourself. This will be useful to you in the future to keep track of changes when you are writing papers. I also recommend creating a Bitbucket account, which has similar functionality, but allows you to have unlimited private repositories for personal use. Additionally, I suggest you read Gentzkow & Shapiro's Code and Data for the Social Sciences: A Practitioner’s Guide to familiarize yourself with good practices in coding and statistical analysis. We will cover additional topics in class.

MacOS

Homebrew

On MacOS you can use Homebrew, which is an excellent tool for installing all kinds of open source software. First, download and install XCode (from the Apple store) and command line tools (option within XCode or in a terminal execute xcode-select --install). Second, install Homebrew

/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

Third, install some of the software we will use, including qgis, MacTeX, R, Rstudio

brew tap osgeo/osgeo4mac
brew install qgis
brew install mactex
brew install r
brew install rstudio

There are many tools that can be installed via Homebrew whenever needed. Try it out!

iTerm2

I highly recommend using iTerm2 instead of the terminal that comes with OSX. It is custumizable and very powerful.

Installing QGIS

Download and install QGIS from their website.

Text Editor

You should have a good text editor, which hopefully has syntax highlighting for various languages, especially for python, R and Stata. I personally recommend atom, which is free, custimizable and very flexible.

LaTeX/LyX

Install the LaTeX distribution for your system from the LaTeX website. If LyX is not installed with your distribution and you want to use a WSIWYG front for LaTeX, download and install LyX here.

Another option is to use tectonic, which is supposed to install a smaller version of the TeX distributon and download only packages that you require. You can install it using conda into your GeoPython environment.

To get started with LaTeX check out Overleaf's Introduction or some of its other resources to get started.

Some of my templates for LaTeX:

You can find more templates here or here.

Markdown Editor

It is useful to have a markdown editor to edit files for Github as well as generating websites like this one. Here are a few options:

More information on markdown at Markdown Guide.

Stata Packages

It is useful to keep a list your Stata packages in case you need to reinstall or replicate your environment somewhere else. For this you can use adolist

net install adolist, from(http://fmwww.bc.edu/RePEc/bocode/a)

To create a list of your packages

 adolist store mystatapacks_latest.pkl

To install the packages from the list

adolist install mystatapacks_latest.pkl

You can update an existing list by

 adolist update mystatapacks_latest.pkl

Here is my latest mystatapacks_latest.pkl in case you want to have some useful packges installed.

Installing (I)Python & Jupyter

The easiest and most convenient way to install a working version of IPython with all the required packages and tools is using Continuum's Anaconda Distribution. You can install following the instructions in that website, or if you can just run this script (Mac/Linux). After installing the latest version of Anaconda, add the Anaconda/bin directory to your PATH variable.

In Windows you may need to install some version of Visual C++ (e.g. for Python-2.7 you need Visual C++ 9.0 otherwise choose the most up-to-date version) and Microsoft MPI.

In MACOS download and install XCode (from the Apple store) and command line tools (option within XCode or in a terminal execute xcode-select --install).

Since the basic installation may not have all the packages you need and it is always best to keep an environment for your work, after installation:

A. (Recommended) In a terminal window execute

conda create --name GeoPython39env -c conda-forge -c r -c mro --override-channels python=3.9 georasters geopandas pandas spatialpandas statsmodels xlrd networkx ipykernel ipyparallel ipython ipython_genutils ipywidgets jupyter jupyterlab kiwisolver matplotlib-base matplotlib scikit-image scikit-learn scipy seaborn geoplot geopy geotiff pycountry nb_conda_kernels stata_kernel nltk ipympl

or download this file and import it into the Anconda Navigator (see here).

If you want a very complete environment

conda create --name GeoPython39env -c conda-forge -c r -c mro --override-channels python=3.9 asciitree autopep8 babel basemap beautifulsoup4 blas bokeh bzip2 cartopy colorcet curl cython cytoolz dask dask-core datashader datashape descartes dill docutils earthengine-api fiona gensim geographiclib geopandas geoplot geopy geotiff google-api-python-client google-auth google-auth-httplib2 georasters imageio imagesize ipykernel ipyparallel ipython ipython_genutils ipywidgets jupyter jupyterlab kiwisolver matplotlib-base matplotlib markdown networkx nose numba numpy openpyxl pandas pandas-datareader pandoc pandocfilters pandana pelican pycountry pymc3 pysal urbanaccess rasterstats r r-base r-irkernel statsmodels sympy rpy2 rasterio r-tidyr r-tibble r-dplyr rise seaborn scikit-allel scikit-bio scikit-image scikit-learn scipy seaborn swifter theano unidecode urllib3 wikipedia xlrd jupyter_contrib_nbextensions rstudio camelot-py html5lib ghostscript plotnine nodejs nb_conda_kernels stata_kernel libgfortran-ng nltk ipympl

Next execute 

conda activate GeoPython39env

pip install linearmodels git+https://github.com/jeetsukumaran/DendroPy.git git+https://github.com/dmsul/econtools ecopy facebook-business facebook-sdk fontawesome-markdown geocoder geonamescache git+https://github.com/ozak/googledrivedownloader googletrans ipystata isodate isounidecode latexcodec ldpred linearmodels lingpy mccabe ordered-set plinkio pyarrow pybtex pycldf pycountry pydocstyle pyface pyflakes pylatex pyparsing pyreadstat pytest-cov python-jsonrpc-server python-language-server pytrends quilt ratelim readme-renderer rfc3986 rope simpledbf simplejson smartypants snowballstemmer snuggs stata-kernel tabulate tex2ipy texsoup toolz traits traitsui twine typogrify uritemplate w3lib webencodings wrapt xlwt yapf modin stargazer covid COVID19Py pandasql mplleaflet scikit-gstat binsreg

conda update --all -c conda-forge -c r -c mro

python -m stata_kernel.install
conda install -c conda-forge nodejs -y
jupyter labextension install jupyterlab-stata-highlight
jupyter labextension install verdant-history
ipcluster nbextension enable
jupyter contrib nbextension install --sys-prefix
jupyter labextension install verdant-history
jupyter nbextension enable nbTranslate/main
r -e "IRkernel::installspec()"

ipcluster nbextension enable

This should create an environment with most of the packages we need. We can always install others down the road.

Windows users will need to follow additional instructions to have the Stata kernel working.

B. (Generally not recommended) download one of the following scripts

and execute

conda update conda
conda-env create -f GeoPython3env2019B_nobuilds.yml
# Or uncomment one of these if you want to install the older versions
#conda-env create -f GeoPython2env.yml
#conda-env create -f GeoPython3env.yml

MacOS

Instead of the previous options, you can download this file and execute

conda create --name GeoPython3env --file GeoPython3env-spec-file.txt

After Installation is complete

This should create an environment with all the packages you will require for this course (and more!).

To start using one of the environment you will need to exectute the following command

conda activate GeoPython39env

or for Python 2.x

source activate GeoPython2env

I would suggest using GeoPython38env, since it is the latest and most up-to-date version (I'll try to keep adding new versions as time passes).

Parallel Computing

One of the advantages of Jupyter Notebooks is that they allow you to work (very easily) with multiple processors using ipyparallel. Once you have ipyparallel installed (automatically done for you with the scritps above), you will need to execute the following code once

ipcluster nbextension enable

Stata Kernel

We can use Stata within Jupyter Notebooks with the Stata kernel. I recommend you make sure it is installed and you have highlighting. To install (in case you do not use the YAML environment file above), execute

pip install stata_kernel
python -m stata_kernel.install

and then 

conda install -c conda-forge nodejs -y
jupyter labextension install jupyterlab-stata-highlight

Windows users will need to follow additional instructions to have the Stata kernel working.

R kernel

We can also use R within Jupyter Notebooks with the R kernel. The best way to install it (in case you do not use the YAML environment file above) is to use conda by executing

conda install -c conda-forge -c r r-irkernel

You can also install R and R packages by using conda. Simply execute

conda install -c conda-forge -c r r
conda install -c conda-forge -c r r-PACKAGE_NAME

For the R-kernel to be available you need to open R and execute

IRkernel::installspec()

Running Stata or R in Python

We can also use Stata or R directly within Python. You only need to use the %magic for each after installing the required packages. For example for Stata you need ipystata. Install by executing

pip install ipystata

Then in jupyter execute

import ipystata 
from ipystata.config import config_stata
config_stata('Path to your Stata executable')

Once configured you can use the %%stata magic. E.g.

In[1]: import ipystata  
In[2]: %%stata  
        display "Hello, I am printed in Stata."

More info in the ipystata website.

Similarly, we can use R using rpy2. Install by executing

conda install -c conda-forge -c r rpy2

Then in jupyter execute

In [1]: %load_ext rpy2.ipython
In [2]: %%R

        R.version

More info in the rpy2 website.

Notebooks

Try it out online

If you want to try out some of the tools and play around without installing or if installation fails, use this Jupyter Notebook binder or the JupyterLab binder.

Binder Binder Binder