We will be using (I)Python as our programming language and QGIS for basic GIS exploration. In order to use some of the material available on this website and to share your material with others you should create a GitHub account for yourself. This will be useful to you in the future to keep track of changes when you are writing papers. I also recommend creating a Bitbucket account, which has similar functionality, but allows you to have unlimited private repositories for personal use. Additionally, I suggest you read Gentzkow & Shapiro's Code and Data for the Social Sciences: A Practitioner’s Guide to familiarize yourself with good practices in coding and statistical analysis. We will cover additional topics in class.
MacOS
Homebrew
On MacOS
you can use Homebrew, which is an excellent tool for installing all kinds of open source software. First, download and install XCode (from the Apple store) and command line tools (option within XCode or in a terminal execute xcode-select --install
). Second, install Homebrew
/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
Third, install some of the software we will use, including qgis
, MacTeX
, R
, Rstudio
brew tap osgeo/osgeo4mac
brew install qgis
brew install mactex
brew install r
brew install rstudio
There are many tools that can be installed via Homebrew
whenever needed. Try it out!
iTerm2
I highly recommend using iTerm2 instead of the terminal that comes with OSX
. It is custumizable and very powerful.
Installing QGIS
Download and install QGIS from their website.
Text Editor
You should have a good text editor, which hopefully has syntax highlighting for various languages, especially for python
, R
and Stata
. I personally recommend atom, which is free, custimizable and very flexible.
LaTeX/LyX
Install the LaTeX
distribution for your system from the LaTeX website. If LyX
is not installed with your distribution and you want to use a WSIWYG front for LaTeX
, download and install LyX here.
Another option is to use tectonic, which is supposed to install a smaller version of the TeX
distributon and download only packages that you require. You can install it using conda
into your GeoPython
environment.
To get started with LaTeX
check out Overleaf's Introduction or some of its other resources to get started.
Some of my templates for LaTeX
:
You can find more templates here or here.
Markdown Editor
It is useful to have a markdown editor to edit files for Github
as well as generating websites like this one. Here are a few options:
- MacDown free and opensource for MacOS.
- typora works on MacOS, WIndows and Linux (free beta)
- atom Markdown Preview free and open source for any OS that handles atom
- dillinger online and open source
- stackedit online and open source
More information on markdown at Markdown Guide.
Stata Packages
It is useful to keep a list your Stata packages in case you need to reinstall or replicate your environment somewhere else. For this you can use adolist
net install adolist, from(http://fmwww.bc.edu/RePEc/bocode/a)
To create a list of your packages
adolist store mystatapacks_latest.pkl
To install the packages from the list
adolist install mystatapacks_latest.pkl
You can update an existing list by
adolist update mystatapacks_latest.pkl
Here is my latest mystatapacks_latest.pkl
in case you want to have some useful packges installed.
Installing (I)Python & Jupyter
The easiest and most convenient way to install a working version of IPython with all the required packages and tools is using Continuum's Anaconda Distribution. You can install following the instructions in that website, or if you can just run this script (Mac/Linux). After installing the latest version of Anaconda, add the Anaconda/bin
directory to your PATH
variable.
In Windows
you may need to install some version of Visual C++ (e.g. for Python-2.7
you need Visual C++ 9.0 otherwise choose the most up-to-date version) and Microsoft MPI.
In MACOS
download and install XCode
(from the Apple store) and command line tools (option within XCode
or in a terminal execute xcode-select --install
).
Since the basic installation may not have all the packages you need and it is always best to keep an environment for your work, after installation:
A. (Recommended) In a terminal window execute
conda create --name GeoPython39env -c conda-forge -c r -c mro --override-channels python=3.9 georasters geopandas pandas spatialpandas statsmodels xlrd networkx ipykernel ipyparallel ipython ipython_genutils ipywidgets jupyter jupyterlab kiwisolver matplotlib-base matplotlib scikit-image scikit-learn scipy seaborn geoplot geopy geotiff pycountry nb_conda_kernels stata_kernel nltk ipympl
or download this file and import it into the Anconda Navigator (see here).
If you want a very complete environment
conda create --name GeoPython39env -c conda-forge -c r -c mro --override-channels python=3.9 asciitree autopep8 babel basemap beautifulsoup4 blas bokeh bzip2 cartopy colorcet curl cython cytoolz dask dask-core datashader datashape descartes dill docutils earthengine-api fiona gensim geographiclib geopandas geoplot geopy geotiff google-api-python-client google-auth google-auth-httplib2 georasters imageio imagesize ipykernel ipyparallel ipython ipython_genutils ipywidgets jupyter jupyterlab kiwisolver matplotlib-base matplotlib markdown networkx nose numba numpy openpyxl pandas pandas-datareader pandoc pandocfilters pandana pelican pycountry pymc3 pysal urbanaccess rasterstats r r-base r-irkernel statsmodels sympy rpy2 rasterio r-tidyr r-tibble r-dplyr rise seaborn scikit-allel scikit-bio scikit-image scikit-learn scipy seaborn swifter theano unidecode urllib3 wikipedia xlrd jupyter_contrib_nbextensions rstudio camelot-py html5lib ghostscript plotnine nodejs nb_conda_kernels stata_kernel libgfortran-ng nltk ipympl
Next execute
conda activate GeoPython39env
pip install linearmodels git+https://github.com/jeetsukumaran/DendroPy.git git+https://github.com/dmsul/econtools ecopy facebook-business facebook-sdk fontawesome-markdown geocoder geonamescache git+https://github.com/ozak/googledrivedownloader googletrans ipystata isodate isounidecode latexcodec ldpred linearmodels lingpy mccabe ordered-set plinkio pyarrow pybtex pycldf pycountry pydocstyle pyface pyflakes pylatex pyparsing pyreadstat pytest-cov python-jsonrpc-server python-language-server pytrends quilt ratelim readme-renderer rfc3986 rope simpledbf simplejson smartypants snowballstemmer snuggs stata-kernel tabulate tex2ipy texsoup toolz traits traitsui twine typogrify uritemplate w3lib webencodings wrapt xlwt yapf modin stargazer covid COVID19Py pandasql mplleaflet scikit-gstat binsreg
conda update --all -c conda-forge -c r -c mro
python -m stata_kernel.install
conda install -c conda-forge nodejs -y
jupyter labextension install jupyterlab-stata-highlight
jupyter labextension install verdant-history
ipcluster nbextension enable
jupyter contrib nbextension install --sys-prefix
jupyter labextension install verdant-history
jupyter nbextension enable nbTranslate/main
r -e "IRkernel::installspec()"
ipcluster nbextension enable
This should create an environment with most of the packages we need. We can always install others down the road.
Windows
users will need to follow additional instructions to have the Stata
kernel working.
B. (Generally not recommended) download one of the following scripts
- GeoPython38env
- GeoPython37env
- GeoPython35env (older Python 3.5 environment)
- GeoPython3env old (older Python 3 environment)
- GeoPython2env (older Python 2 environment)
and execute
conda update conda
conda-env create -f GeoPython3env2019B_nobuilds.yml
# Or uncomment one of these if you want to install the older versions
#conda-env create -f GeoPython2env.yml
#conda-env create -f GeoPython3env.yml
MacOS
Instead of the previous options, you can download this file and execute
conda create --name GeoPython3env --file GeoPython3env-spec-file.txt
After Installation is complete
This should create an environment with all the packages you will require for this course (and more!).
To start using one of the environment you will need to exectute the following command
conda activate GeoPython39env
or for Python 2.x
source activate GeoPython2env
I would suggest using GeoPython38env
, since it is the latest and most up-to-date version (I'll try to keep adding new versions as time passes).
Parallel Computing
One of the advantages of Jupyter Notebooks
is that they allow you to work (very easily) with multiple processors using ipyparallel
. Once you have ipyparallel
installed (automatically done for you with the scritps above), you will need to execute the following code once
ipcluster nbextension enable
Stata Kernel
We can use Stata
within Jupyter Notebooks with the Stata kernel. I recommend you make sure it is installed and you have highlighting. To install (in case you do not use the YAML
environment file above), execute
pip install stata_kernel
python -m stata_kernel.install
and then
conda install -c conda-forge nodejs -y
jupyter labextension install jupyterlab-stata-highlight
Windows
users will need to follow additional instructions to have the Stata
kernel working.
R kernel
We can also use R
within Jupyter Notebooks with the R kernel. The best way to install it (in case you do not use the YAML
environment file above) is to use conda
by executing
conda install -c conda-forge -c r r-irkernel
You can also install R
and R
packages by using conda
. Simply execute
conda install -c conda-forge -c r r
conda install -c conda-forge -c r r-PACKAGE_NAME
For the R-kernel
to be available you need to open R
and execute
IRkernel::installspec()
Running Stata or R in Python
We can also use Stata
or R
directly within Python
. You only need to use the %magic
for each after installing the required packages. For example for Stata
you need ipystata
. Install by executing
pip install ipystata
Then in jupyter
execute
import ipystata
from ipystata.config import config_stata
config_stata('Path to your Stata executable')
Once configured you can use the %%stata
magic. E.g.
In[1]: import ipystata
In[2]: %%stata
display "Hello, I am printed in Stata."
More info in the ipystata
website.
Similarly, we can use R
using rpy2
. Install by executing
conda install -c conda-forge -c r rpy2
Then in jupyter execute
In [1]: %load_ext rpy2.ipython
In [2]: %%R
R.version
More info in the rpy2
website.
Notebooks
- Notebook 1: Introduction (html) (ipynb)
- Notebook 2: Introduction to CGE (html) (ipynb)
- Notebook 3: Dynamic Programming in Python (html) (ipynb)
- Notebook 4: Faster Dynamic Programming with Numba (html) ipynb
- Notebook 5: Economic Data Analysis (html) (ipynb)
- Notebook 6: GIS with QGIS (html) (ipynb)
- Notebook 7: GIS with Python (Geometries) (html) (ipynb)
- Notebook 8: GIS with Python 2 (Rasters) (html) (ipynb)
- Notebook 9: GIS with Python 3 (Data Munging) (html) (ipynb)
- Notebook 10: Stata Jupyter Notebooks (Stata kernel) (html) (ipynb)
- Notebook 11: Stata in a Python Jupyter Notebook (ipystata) (html) (ipynb)
Try it out online
If you want to try out some of the tools and play around without installing or if installation fails, use this Jupyter Notebook binder or the JupyterLab binder.