issues

Table actions

5 rows where "created_at" is on date 2019-09-03, repo = 206156866 and user = 9599 sorted by updated_at descending

✎ View and edit SQL

This data as json, CSV (advanced)

Suggested facets: created_at (date), updated_at (date), closed_at (date)

state 2

type 1

repo 1

  • twitter-to-sqlite · 5 ✖
id node_id number title user state locked assignee milestone comments created_at updated_at ▲ closed_at author_association pull_request body repo type active_lock_reason performed_via_github_app reactions draft state_reason
488833975 MDU6SXNzdWU0ODg4MzM5NzU= 3 Command for running a search and saving tweets for that search simonw 9599 closed 0     6 2019-09-03T21:29:56Z 2019-11-04T05:31:56Z 2019-11-04T05:31:16Z MEMBER   
$ twitter-to-sqlite search dogsheep
twitter-to-sqlite 206156866 issue     
{
    "url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/3/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
488835586 MDU6SXNzdWU0ODg4MzU1ODY= 4 Command for importing data from a Twitter Export file simonw 9599 closed 0     2 2019-09-03T21:34:13Z 2019-10-11T06:45:02Z 2019-10-11T06:45:02Z MEMBER  

Twitter lets you export all of your data as an archive file: https://twitter.com/settings/your_twitter_data

A command for importing this data into SQLite would be extremely useful.

$ twitter-to-sqlite import twitter.db path-to-archive.zip
twitter-to-sqlite 206156866 issue     
{
    "url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/4/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
488833136 MDU6SXNzdWU0ODg4MzMxMzY= 1 Imported followers should go in "users", relationships in "following" simonw 9599 closed 0     0 2019-09-03T21:27:37Z 2019-09-04T20:23:04Z 2019-09-04T20:23:04Z MEMBER  

Right now twitter-to-sqlite followers dumps everything in a followers table, and doesn't actually record which account they are following!

It should instead save them all in a global users table and then set up m2m relationships in a following table. This also means it should create a record for the specified user in order to record both sides of each relationship.

 twitter-to-sqlite 206156866 issue     
{
    "url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/1/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
488833698 MDU6SXNzdWU0ODg4MzM2OTg= 2 "twitter-to-sqlite user-timeline" command for pulling tweets by a specific user simonw 9599 closed 0     3 2019-09-03T21:29:12Z 2019-09-04T20:02:11Z 2019-09-04T20:02:11Z MEMBER  

Twitter only allows up to 3,200 tweets to be retrieved from https://developer.twitter.com/en/docs/tweets/timelines/api-reference/get-statuses-user_timeline.html

I'm going to do:

$ twitter-to-sqlite tweets simonw
twitter-to-sqlite 206156866 issue     
{
    "url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/2/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
488874815 MDU6SXNzdWU0ODg4NzQ4MTU= 5 Write tests that simulate the Twitter API simonw 9599 open 0     1 2019-09-03T23:55:35Z 2019-09-03T23:56:28Z   MEMBER  

I can use betamax for this: https://pypi.org/project/betamax/

 twitter-to-sqlite 206156866 issue     
{
    "url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/5/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options: 

CREATE TABLE [issues] (
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [number] INTEGER,
   [title] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [state] TEXT,
   [locked] INTEGER,
   [assignee] INTEGER REFERENCES [users]([id]),
   [milestone] INTEGER REFERENCES [milestones]([id]),
   [comments] INTEGER,
   [created_at] TEXT,
   [updated_at] TEXT,
   [closed_at] TEXT,
   [author_association] TEXT,
   [pull_request] TEXT,
   [body] TEXT,
   [repo] INTEGER REFERENCES [repos]([id]),
   [type] TEXT
, [active_lock_reason] TEXT, [performed_via_github_app] TEXT, [reactions] TEXT, [draft] INTEGER, [state_reason] TEXT);
CREATE INDEX [idx_issues_repo]
                ON [issues] ([repo]);
CREATE INDEX [idx_issues_milestone]
                ON [issues] ([milestone]);
CREATE INDEX [idx_issues_assignee]
                ON [issues] ([assignee]);
CREATE INDEX [idx_issues_user]
                ON [issues] ([user]);