theenglishway-corp/twhatter: A simple scraper for Twitter (requires Python >= 3.6) @ 50f7a9f4a2bbd70e60cde8e95d9a5b76a3b9f1df

A simple scraper for Twitter (requires Python >= 3.6)

theenglishway (time) 50f7a9f4a2 Add JSON output		vor 7 Jahren
.github	4935b1edbf Initial commit	vor 7 Jahren
docs	4935b1edbf Initial commit	vor 7 Jahren
tests	e4a4d93197 Stop client loop as early as possible	vor 7 Jahren
twhatter	50f7a9f4a2 Add JSON output	vor 7 Jahren
.editorconfig	4935b1edbf Initial commit	vor 7 Jahren
.gitignore	4935b1edbf Initial commit	vor 7 Jahren
.travis.yml	4935b1edbf Initial commit	vor 7 Jahren
AUTHORS.rst	4935b1edbf Initial commit	vor 7 Jahren
CONTRIBUTING.rst	4935b1edbf Initial commit	vor 7 Jahren
HISTORY.rst	4935b1edbf Initial commit	vor 7 Jahren
LICENSE	4935b1edbf Initial commit	vor 7 Jahren
MANIFEST.in	4935b1edbf Initial commit	vor 7 Jahren
Makefile	4935b1edbf Initial commit	vor 7 Jahren
Pipfile	9539167374 Use library for user-agent	vor 7 Jahren
Pipfile.lock	9539167374 Use library for user-agent	vor 7 Jahren
README.rst	7a45c6a99f Update README	vor 7 Jahren
setup.cfg	f22591146f Fix installation on Python < 3.7	vor 7 Jahren
setup.py	4935b1edbf Initial commit	vor 7 Jahren
tox.ini	4935b1edbf Initial commit	vor 7 Jahren

		
				README.rst
			
				========
Twhatter
========

A simple Python scraper for Twitter.

Motivation
----------

Twitter's API `terms and conditions `_
have become very demanding in May 2018. Inspired by other attempts, I have
put together yet another twitter scraper that uses a simple HTTP client instead
of the developer API, and allows retrieving any data that can be accessed in an
anonymous browsing session.

This is mostly an attempt for me to produce some clean, functional and
maintainable Python code. I have especially focused on a clean separation
between data retrieval and output, which should allow to export data in any
format.

And why that terrible name ? Simple, "WHAT's going on TWITTER ?" => TWHATTER !

Features
--------

At the moment, this utility only provides a command-line to interact with it.

Anonymous client
****************

- Get any user's full timeline.
- Get any user's profile data.

Data output
***********

All scraped information can either be displayed on the terminal or stored into
a local database.

Installation
------------

Installation requires Python >= 3.6. ::

    $ pip install --user git+https://code.theenglishway.eu/theenglishway-corp/twhatter

You then have to ensure that `~/.local/bin` in your `$PATH` or call
`~/.local/bin/twhatter` instead of `twhatter` in the following examples

Usage
-----

Display some user's tweets ::

    $ twhatter timeline realDonaldTrump --limit 10
    
Display their profile information ::

    $ twhatter profile realDonaldTrump
    User(id=25073877, screen_name='Donald J. Trump', join_date=datetime.datetime(2009, 3, 18, 0, 0), tweets_nb=40183, following_nb=45, followers_nb=57144827, likes_nb=7)

Put them into a local database (by default in /tmp/db.sqlite) ::

    $ twhatter db timeline realDonaldTrump

Open a session on the local database and make queries with SQLAlchemy ::

    $ twhatter db shell

    In [1]: session.query(Tweet).all()
    Out[1]:
    [                         
Tests
-----

Unit tests are provided. From the root directory in a freshly-cloned repository
and with a clean virtual environment, they can be run with ::

    $ pytest

Useful links
------------

* `Raymond Hettinger's excellent talk about Python 3.7's Dataclasses `_

Other scrapers that might fit your needs
****************************************

In Python :

* `twint `_
* `twitterscraper `_
* `twitter-scraper `_

In Javascript:

* `scrape-twitter `_