issue_comments

In testing this collects a LOT of data. I'm going to skip tracking favourites_count and statuses_count and just track followers, friends and listed instead.

Actually a single table with a type integer ID referencing a count_history_types table would better match the way I implemented the since_ids table:

https://github.com/dogsheep/twitter-to-sqlite/blob/4b6c8d8c1cc6fefdb566ec8506157133f47c569a/twitter_to_sqlite/utils.py#L331-L341

In which case the compound primary key would be type, user, datetime

I want the datetime value to look like 2020-04-01T03:34:58+00:00 (the format returned by the Twitter API which I am storing in other tables at the moment). ```

datetime.utcnow().isoformat().split('.')[0] + '+00:00' '2020-04-01T03:49:52+00:00' ```

I'm happy with the recent work I did on this.

OK, fix is applied to everything now.

I've got this working for the user-timeline command.

The tricky thing here is thinking about the interaction between the recorded since_id and a desire to run the initial import.

The first time you run twitter-to-sqlite user-timeline db.db username we want to fetch as many tweets from that user as possible - probably around 3,200 before the API limitations cut us off.

We need to record the maximum ID from those as the since_id - which we will see on the very first page we paginate through. That way next time we run the command with --since we will only fetch new tweets.

But what happens if our initial import is cancelled after only a few tweets? We risk never pulling in the rest of the tweets.

Not sure if I need to solve this at all or if I should instead trust users to run the command a second time without --since if they think they didn't retrieve anything the first time.

I had considered letting --stop_after= over-ride --since but that doesn't actually make sense - if you send a since_id to the Twitter API you'll never get back more tweets than exist after that ID, so the --stop_after would not make a meaningful difference.