issues

Table actions

4 rows where comments = 4 and repo = 206156866 sorted by updated_at descending

✎ View and edit SQL

This data as json, CSV (advanced)

Suggested facets: user, author_association, created_at (date), updated_at (date), closed_at (date)

type 1

state 1

repo 1

  • twitter-to-sqlite · 4 ✖
id node_id number title user state locked assignee milestone comments created_at updated_at ▲ closed_at author_association pull_request body repo type active_lock_reason performed_via_github_app reactions draft state_reason
490803176 MDU6SXNzdWU0OTA4MDMxNzY= 8 --sql and --attach options for feeding commands from SQL queries simonw 9599 closed 0     4 2019-09-08T20:35:49Z 2020-03-20T23:13:01Z 2020-03-20T23:13:01Z MEMBER  

Say you want to fetch Twitter profiles for a list of accounts that are stored in another database:

$ twitter-to-sqlite users-lookup users.db --attach attending.db \
    --sql "select Twitter from attending.attendes where Twitter is not null"

The SQL query you feed in is expected to return a list of screen names suitable for processing further by the command.

Should be supported by all three of:

  • [x] twitter-to-sqlite users-lookup
  • [x] twitter-to-sqlite user-timeline
  • [x] twitter-to-sqlite followers and friends

The --attach option allows other SQLite databases to be attached to the connection. Without it the SQL query will have to read from the single attached database.

 twitter-to-sqlite 206156866 issue     
{
    "url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/8/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
518725064 MDU6SXNzdWU1MTg3MjUwNjQ= 29 `import` command fails on empty files jacobian 21148 closed 0     4 2019-11-06T20:34:26Z 2019-11-09T20:33:38Z 2019-11-09T19:36:36Z CONTRIBUTOR  

If a file in the export is empty (in my case it was account-suspensions.js), twitter-to-sqlite import fails:

$ twitter-to-sqlite import twitter.db ~/Downloads/twitter-2019-11-06-926f4f3be4b3b1fcb1aa387c40cd14f7c8aaf9bbcdb2d78ac14d9989add501bb.zip Traceback (most recent call last): File "/Users/jacob/Library/Caches/pypoetry/virtualenvs/jkm-dogsheep-ezLnyXZS-py3.7/bin/twitter-to-sqlite", line 10, in <module> sys.exit(cli()) File "/Users/jacob/Library/Caches/pypoetry/virtualenvs/jkm-dogsheep-ezLnyXZS-py3.7/lib/python3.7/site-packages/click/core.py", line 764, in __call__ return self.main(*args, **kwargs) File "/Users/jacob/Library/Caches/pypoetry/virtualenvs/jkm-dogsheep-ezLnyXZS-py3.7/lib/python3.7/site-packages/click/core.py", line 717, in main rv = self.invoke(ctx) File "/Users/jacob/Library/Caches/pypoetry/virtualenvs/jkm-dogsheep-ezLnyXZS-py3.7/lib/python3.7/site-packages/click/core.py", line 1137, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/Users/jacob/Library/Caches/pypoetry/virtualenvs/jkm-dogsheep-ezLnyXZS-py3.7/lib/python3.7/site-packages/click/core.py", line 956, in invoke return ctx.invoke(self.callback, **ctx.params) File "/Users/jacob/Library/Caches/pypoetry/virtualenvs/jkm-dogsheep-ezLnyXZS-py3.7/lib/python3.7/site-packages/click/core.py", line 555, in invoke return callback(*args, **kwargs) File "/Users/jacob/Library/Caches/pypoetry/virtualenvs/jkm-dogsheep-ezLnyXZS-py3.7/lib/python3.7/site-packages/twitter_to_sqlite/cli.py", line 627, in import_ archive.import_from_file(db, filename, content) File "/Users/jacob/Library/Caches/pypoetry/virtualenvs/jkm-dogsheep-ezLnyXZS-py3.7/lib/python3.7/site-packages/twitter_to_sqlite/archive.py", line 224, in import_from_file db[table_name].upsert_all(rows, hash_id="pk") File "/Users/jacob/Library/Caches/pypoetry/virtualenvs/jkm-dogsheep-ezLnyXZS-py3.7/lib/python3.7/site-packages/sqlite_utils/db.py", line 1113, in upsert_all extracts=extracts, File "/Users/jacob/Library/Caches/pypoetry/virtualenvs/jkm-dogsheep-ezLnyXZS-py3.7/lib/python3.7/site-packages/sqlite_utils/db.py", line 980, in insert_all first_record = next(records) StopIteration

This appears to be because db.upsert_all is called with no rows -- I think?

I hacked around this by modifying import_from_file to have an if rows: clause:

for table, rows in to_insert.items(): if rows: table_name = "archive_{}".format(table.replace("-", "_")) ...

I'm happy to work up a real PR if that's the right approach, but I'm not sure it is.

 twitter-to-sqlite 206156866 issue     
{
    "url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/29/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
514459062 MDU6SXNzdWU1MTQ0NTkwNjI= 27 retweets-of-me command simonw 9599 closed 0     4 2019-10-30T07:43:01Z 2019-11-03T01:12:58Z 2019-11-03T01:12:58Z MEMBER   twitter-to-sqlite 206156866 issue     
{
    "url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/27/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed
505928530 MDU6SXNzdWU1MDU5Mjg1MzA= 18 Command to import home-timeline simonw 9599 closed 0     4 2019-10-11T15:47:54Z 2019-10-11T16:51:33Z 2019-10-11T16:51:12Z MEMBER  

Feature request: https://twitter.com/johankj/status/1182563563136868352

Would it be possible to save all tweets in my timeline from the last X days? I would love to see how big a percentage some users are of my daily timeline as a metric on whether I should unfollow them/move them to a list.

 twitter-to-sqlite 206156866 issue     
{
    "url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/18/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options: 

CREATE TABLE [issues] (
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [number] INTEGER,
   [title] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [state] TEXT,
   [locked] INTEGER,
   [assignee] INTEGER REFERENCES [users]([id]),
   [milestone] INTEGER REFERENCES [milestones]([id]),
   [comments] INTEGER,
   [created_at] TEXT,
   [updated_at] TEXT,
   [closed_at] TEXT,
   [author_association] TEXT,
   [pull_request] TEXT,
   [body] TEXT,
   [repo] INTEGER REFERENCES [repos]([id]),
   [type] TEXT
, [active_lock_reason] TEXT, [performed_via_github_app] TEXT, [reactions] TEXT, [draft] INTEGER, [state_reason] TEXT);
CREATE INDEX [idx_issues_repo]
                ON [issues] ([repo]);
CREATE INDEX [idx_issues_milestone]
                ON [issues] ([milestone]);
CREATE INDEX [idx_issues_assignee]
                ON [issues] ([assignee]);
CREATE INDEX [idx_issues_user]
                ON [issues] ([user]);