issue_comments

Table actions

9 rows where author_association = "MEMBER" and "created_at" is on date 2019-10-17 sorted by updated_at descending

✖
✖
✖

✎ View and edit SQL

This data as json, CSV (advanced)

Suggested facets: issue_url, created_at (date), updated_at (date)

user 1

author_association 1

  • MEMBER · 9 ✖
id html_url issue_url node_id user created_at updated_at ▲ author_association body reactions issue performed_via_github_app
543290744 https://github.com/dogsheep/twitter-to-sqlite/issues/3#issuecomment-543290744 https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/3 MDEyOklzc3VlQ29tbWVudDU0MzI5MDc0NA== simonw 9599 2019-10-17T17:57:14Z 2019-10-17T17:57:14Z MEMBER

I have a working command now. I'm going to ship it early because it could do with some other people trying it out.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Command for running a search and saving tweets for that search 488833975  
543273540 https://github.com/dogsheep/twitter-to-sqlite/issues/3#issuecomment-543273540 https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/3 MDEyOklzc3VlQ29tbWVudDU0MzI3MzU0MA== simonw 9599 2019-10-17T17:12:51Z 2019-10-17T17:12:51Z MEMBER

Just importing tweets here isn't enough - how are we supposed to know which tweets were imported by which search?

So I think the right thing to do here is to also create a search_runs table, which records each individual run of this tool (with a timestamp and the search terms used). Then have a search_runs_tweets m2m table which shows which Tweets were found by that search.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Command for running a search and saving tweets for that search 488833975  
543271000 https://github.com/dogsheep/twitter-to-sqlite/issues/10#issuecomment-543271000 https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/10 MDEyOklzc3VlQ29tbWVudDU0MzI3MTAwMA== simonw 9599 2019-10-17T17:05:59Z 2019-10-17T17:05:59Z MEMBER 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Rethink progress bars for various commands 492297930  
543270714 https://github.com/dogsheep/twitter-to-sqlite/issues/10#issuecomment-543270714 https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/10 MDEyOklzc3VlQ29tbWVudDU0MzI3MDcxNA== simonw 9599 2019-10-17T17:05:16Z 2019-10-17T17:05:16Z MEMBER

https://github.com/pallets/click/blob/716a5be90f56ce6cd506bb53d5739d09374b1636/click/_termui_impl.py#L93 is how Click does this: self.is_hidden = not isatty(self.file) Where isatty is a Click utility function: from ._compat import isatty

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Rethink progress bars for various commands 492297930  
543269396 https://github.com/dogsheep/twitter-to-sqlite/issues/10#issuecomment-543269396 https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/10 MDEyOklzc3VlQ29tbWVudDU0MzI2OTM5Ng== simonw 9599 2019-10-17T17:02:07Z 2019-10-17T17:02:07Z MEMBER

A neat trick that Click does is detecting if an interactive terminal is attached and NOT showing a progress bar if there isn't one. Need to figure out how to do that with tqdm.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Rethink progress bars for various commands 492297930  
543266947 https://github.com/dogsheep/twitter-to-sqlite/issues/25#issuecomment-543266947 https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/25 MDEyOklzc3VlQ29tbWVudDU0MzI2Njk0Nw== simonw 9599 2019-10-17T16:56:06Z 2019-10-17T16:56:06Z MEMBER

I wrote a test that proves that this is a problem. Should be an easy fix though.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Ensure migrations don't accidentally create foreign key twice 508578780  
543265058 https://github.com/dogsheep/twitter-to-sqlite/issues/25#issuecomment-543265058 https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/25 MDEyOklzc3VlQ29tbWVudDU0MzI2NTA1OA== simonw 9599 2019-10-17T16:51:12Z 2019-10-17T16:51:12Z MEMBER

This migration function only runs if there is a table called tweets and the migration has not run before.

I think this can happen if the database has just been freshly created (by a command that fetches the user's user timeline for example) and is then run a SECOND time.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Ensure migrations don't accidentally create foreign key twice 508578780  
543222239 https://github.com/dogsheep/twitter-to-sqlite/issues/23#issuecomment-543222239 https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/23 MDEyOklzc3VlQ29tbWVudDU0MzIyMjIzOQ== simonw 9599 2019-10-17T15:12:33Z 2019-10-17T15:12:33Z MEMBER

Migrations will run only if you open a database that previously existed (as opposed to opening a brand new empty database).

This means that the first time you run a command against a fresh database, migrations will not run and the migrations table will not be created. The second time you run any command against that database the migrations will execute and populate the migrations table.

This also means that each migration needs to be able to sanity check the database to see if it should run or not. If it should NOT run, it will do nothing but still be marked as having executed by adding to the migrations table.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Extremely simple migration system 508190730  
543217890 https://github.com/dogsheep/twitter-to-sqlite/issues/23#issuecomment-543217890 https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/23 MDEyOklzc3VlQ29tbWVudDU0MzIxNzg5MA== simonw 9599 2019-10-17T15:03:10Z 2019-10-17T15:03:10Z MEMBER

Thinking about this further: the concept of migrations may end up being in direct conflict with the sqlite-utils concept of creating tables on demand the first time they are used - and of creating table schemas automatically to fit the shape of the JSON that is being inserted into them.

I'm going to forge ahead anyway and build this because I think it will be an interesting exploration, but it's very likely this turns out to be a bad idea in the long run!

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 Extremely simple migration system 508190730  

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options: 

CREATE TABLE [issue_comments] (
   [html_url] TEXT,
   [issue_url] TEXT,
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [created_at] TEXT,
   [updated_at] TEXT,
   [author_association] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [issue] INTEGER REFERENCES [issues]([id])
, [performed_via_github_app] TEXT);
CREATE INDEX [idx_issue_comments_issue]
                ON [issue_comments] ([issue]);
CREATE INDEX [idx_issue_comments_user]
                ON [issue_comments] ([user]);