Tricoteuses-Senat

Handle French Sénat's open data

Installation

git clone https://git.en-root.org/tricoteuses/tricoteuses-senat
cd tricoteuses-senat/

Create a .env file to set PostgreSQL database informations and other configuration variables (you can use example.env as a template). Then

npm install

Database creation

Using Docker

docker run --name local-postgres -d -p 5432:5432 -e POSTGRES_PASSWORD=$YOUR_CUSTOM_DB_PASSWORD postgres
# Default Postgres user is postgres
# But scripts require an "opendata" role
docker exec -it local-postgres psql -U postgres -c "CREATE ROLE opendata;"

Usage

Retrieval, cleaning & import of open data from Sénat

mkdir ../senat-data/
npx babel-node --extensions ".ts" --max-old-space-size=4096 -- src/scripts/retrieve_open_data.ts --fetch --unzip --repair-encoding ../senat-data/
npx babel-node --extensions ".ts" -- src/scripts/retrieve_open_data.ts --import ../senat-data/
npx babel-node --extensions ".ts" -- src/scripts/fix_db.ts --updateSchema
npx babel-node --extensions ".ts" -- src/scripts/convert_data.ts ../senat-data/ [--categories DosLeg]

Retrieval of sénateurs' pictures from Sénat's website

npx babel-node --extensions ".ts" -- src/scripts/retrieve_senateurs_photos.ts --fetch ../senat-data/

Retrieval & cleaning of bills from Sénat HTML pages

npx babel-node --extensions ".ts" -- src/scripts/retrieve_textes.ts ../senat-data/

Generation of raw types from SQL schema.

To be run after database has been fixed (src/scripts/fix_db.ts --updateSchema).

npx babel-node --extensions ".ts" -- src/scripts/retrieve_open_data.ts --schema ../senat-data/