issue_comments

Table actions

3 rows where author_association = "MEMBER", "created_at" is on date 2020-04-01 and issue = 590666760 sorted by updated_at descending

✖
✖
✖
✖

✎ View and edit SQL

This data as json, CSV (advanced)

Suggested facets: created_at (date), updated_at (date)

user 1

issue 1

  • --since feature can be confused by retweets · 3 ✖

author_association 1

  • MEMBER · 3 ✖
id html_url issue_url node_id user created_at updated_at ▲ author_association body reactions issue performed_via_github_app
607010634 https://github.com/dogsheep/twitter-to-sqlite/issues/39#issuecomment-607010634 https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/39 MDEyOklzc3VlQ29tbWVudDYwNzAxMDYzNA== simonw 9599 2020-04-01T03:45:16Z 2020-04-01T03:45:16Z MEMBER

OK, fix is applied to everything now.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 --since feature can be confused by retweets 590666760  
607003655 https://github.com/dogsheep/twitter-to-sqlite/issues/39#issuecomment-607003655 https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/39 MDEyOklzc3VlQ29tbWVudDYwNzAwMzY1NQ== simonw 9599 2020-04-01T03:18:00Z 2020-04-01T03:18:00Z MEMBER

I've got this working for the user-timeline command.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 --since feature can be confused by retweets 590666760  
606998669 https://github.com/dogsheep/twitter-to-sqlite/issues/39#issuecomment-606998669 https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/39 MDEyOklzc3VlQ29tbWVudDYwNjk5ODY2OQ== simonw 9599 2020-04-01T02:57:36Z 2020-04-01T02:57:36Z MEMBER

The tricky thing here is thinking about the interaction between the recorded since_id and a desire to run the initial import.

The first time you run twitter-to-sqlite user-timeline db.db username we want to fetch as many tweets from that user as possible - probably around 3,200 before the API limitations cut us off.

We need to record the maximum ID from those as the since_id - which we will see on the very first page we paginate through. That way next time we run the command with --since we will only fetch new tweets.

But what happens if our initial import is cancelled after only a few tweets? We risk never pulling in the rest of the tweets.

Not sure if I need to solve this at all or if I should instead trust users to run the command a second time without --since if they think they didn't retrieve anything the first time.

I had considered letting --stop_after= over-ride --since but that doesn't actually make sense - if you send a since_id to the Twitter API you'll never get back more tweets than exist after that ID, so the --stop_after would not make a meaningful difference.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 --since feature can be confused by retweets 590666760  

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options: 

CREATE TABLE [issue_comments] (
   [html_url] TEXT,
   [issue_url] TEXT,
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [created_at] TEXT,
   [updated_at] TEXT,
   [author_association] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [issue] INTEGER REFERENCES [issues]([id])
, [performed_via_github_app] TEXT);
CREATE INDEX [idx_issue_comments_issue]
                ON [issue_comments] ([issue]);
CREATE INDEX [idx_issue_comments_user]
                ON [issue_comments] ([user]);