{"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/39#issuecomment-607010634", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/39", "id": 607010634, "node_id": "MDEyOklzc3VlQ29tbWVudDYwNzAxMDYzNA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-01T03:45:16Z", "updated_at": "2020-04-01T03:45:16Z", "author_association": "MEMBER", "body": "OK, fix is applied to everything now.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 590666760, "label": "--since feature can be confused by retweets"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/39#issuecomment-607003655", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/39", "id": 607003655, "node_id": "MDEyOklzc3VlQ29tbWVudDYwNzAwMzY1NQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-01T03:18:00Z", "updated_at": "2020-04-01T03:18:00Z", "author_association": "MEMBER", "body": "I've got this working for the `user-timeline` command.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 590666760, "label": "--since feature can be confused by retweets"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/39#issuecomment-606850453", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/39", "id": 606850453, "node_id": "MDEyOklzc3VlQ29tbWVudDYwNjg1MDQ1Mw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-31T20:14:58Z", "updated_at": "2020-04-01T03:03:50Z", "author_association": "MEMBER", "body": "Actually I'll hard-code the population of `since_id_types` to get known ID constants.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 590666760, "label": "--since feature can be confused by retweets"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/39#issuecomment-606998669", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/39", "id": 606998669, "node_id": "MDEyOklzc3VlQ29tbWVudDYwNjk5ODY2OQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-04-01T02:57:36Z", "updated_at": "2020-04-01T02:57:36Z", "author_association": "MEMBER", "body": "The tricky thing here is thinking about the interaction between the recorded since_id and a desire to run the initial import.\r\n\r\nThe first time you run `twitter-to-sqlite user-timeline db.db username` we want to fetch as many tweets from that user as possible - probably around 3,200 before the API limitations cut us off.\r\n\r\nWe need to record the maximum ID from those as the `since_id` - which we will see on the very first page we paginate through. That way next time we run the command with `--since` we will only fetch new tweets.\r\n\r\nBut what happens if our initial import is cancelled after only a few tweets? We risk never pulling in the rest of the tweets.\r\n\r\nNot sure if I need to solve this at all or if I should instead trust users to run the command a second time without `--since` if they think they didn't retrieve anything the first time.\r\n\r\nI had considered letting `--stop_after=` over-ride `--since` but that doesn't actually make sense - if you send a since_id to the Twitter API you'll never get back more tweets than exist after that ID, so the `--stop_after` would not make a meaningful difference.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 590666760, "label": "--since feature can be confused by retweets"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/39#issuecomment-606850008", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/39", "id": 606850008, "node_id": "MDEyOklzc3VlQ29tbWVudDYwNjg1MDAwOA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-31T20:13:59Z", "updated_at": "2020-04-01T00:23:00Z", "author_association": "MEMBER", "body": "Table design for `since_ids` table:\r\n\r\ntype | key | since_id\r\n--- | --- | ---\r\n1 | 124324 | 2347239847293\r\n2 | 99ff9cefff5cbfd804f7cd43e2b27ced8addbe8d | 2125947927344\r\n\r\nPrimary compound key on `(category, key)`\r\n\r\n`type` is also a foreign key to a `since_id_types` table with `id` and `name` columns (probably created using https://sqlite-utils.readthedocs.io/en/stable/python-api.html#working-with-lookup-tables )", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 590666760, "label": "--since feature can be confused by retweets"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/39#issuecomment-606843224", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/39", "id": 606843224, "node_id": "MDEyOklzc3VlQ29tbWVudDYwNjg0MzIyNA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-31T19:59:11Z", "updated_at": "2020-03-31T20:06:32Z", "author_association": "MEMBER", "body": "Or... have a single `since_ids` table to track since values, and have its primary key be a string that looks something like this:\r\n\r\n`user:123145`\r\n`home:23441`\r\n`mentions:23425`\r\n`search:99ff9cefff5cbfd804f7cd43e2b27ced8addbe8d`\r\n\r\nThat last example would use the hash generated here:\r\n\r\nhttps://github.com/dogsheep/twitter-to-sqlite/blob/810cb2af5a175837204389fd7f4b5721f8b325ab/twitter_to_sqlite/cli.py#L792-L808", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 590666760, "label": "--since feature can be confused by retweets"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/39#issuecomment-606844521", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/39", "id": 606844521, "node_id": "MDEyOklzc3VlQ29tbWVudDYwNjg0NDUyMQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-31T20:01:39Z", "updated_at": "2020-03-31T20:01:39Z", "author_association": "MEMBER", "body": "I think `utils.fetch_timeline()` grows a new argument, `since_key`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 590666760, "label": "--since feature can be confused by retweets"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/39#issuecomment-606824992", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/39", "id": 606824992, "node_id": "MDEyOklzc3VlQ29tbWVudDYwNjgyNDk5Mg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-31T19:24:23Z", "updated_at": "2020-03-31T19:24:23Z", "author_association": "MEMBER", "body": "The `--since` option is actually used by four commands:\r\n\r\n* `user-timeline`\r\n* `home-timeline`\r\n* `mentions-timeline`\r\n* `search`\r\n\r\nAll of them use the same `fetch_timeline()` utility function under the hood. I should move the logic that looks up the last `since_id` into that shared function.\r\n\r\nQuestion: should I have a table for each of those four methods or a single table that is used by them all? I'm leaning towards four separate tables.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 590666760, "label": "--since feature can be confused by retweets"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/39#issuecomment-606309165", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/39", "id": 606309165, "node_id": "MDEyOklzc3VlQ29tbWVudDYwNjMwOTE2NQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-30T23:41:31Z", "updated_at": "2020-03-30T23:41:31Z", "author_association": "MEMBER", "body": "I like the separate `user_timeline_since` table solution.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 590666760, "label": "--since feature can be confused by retweets"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/39#issuecomment-606305701", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/39", "id": 606305701, "node_id": "MDEyOklzc3VlQ29tbWVudDYwNjMwNTcwMQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-30T23:30:27Z", "updated_at": "2020-03-30T23:30:27Z", "author_association": "MEMBER", "body": "A better alternative would be to maintain a separate table with the last seen since value for when we ran `user-timeline` for any specific user.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 590666760, "label": "--since feature can be confused by retweets"}, "performed_via_github_app": null}
{"html_url": "https://github.com/dogsheep/twitter-to-sqlite/issues/39#issuecomment-606304837", "issue_url": "https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/39", "id": 606304837, "node_id": "MDEyOklzc3VlQ29tbWVudDYwNjMwNDgzNw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-03-30T23:27:50Z", "updated_at": "2020-03-30T23:29:31Z", "author_association": "MEMBER", "body": "One option would be something like this:\r\n```sql\r\nselect max(id) from tweets\r\nwhere user = ?\r\nand not exists (select id from tweets where retweeted_status = id)\r\nand not exists (select id from tweets where quoted_status = id)\r\nand not exists (select id from tweets where in_reply_to_status_id = id)\r\n```\r\nMight be a good idea to index those columns (after confirming that doing so would indeed speed up the query).", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 590666760, "label": "--since feature can be confused by retweets"}, "performed_via_github_app": null}