issue_comments
10,495 rows sorted by html_url
This data as json, CSV (advanced)
issue >30
- Redesign default .json format 55
- Show column metadata plus links for foreign keys on arbitrary query results 51
- ?_extra= support (draft) 49
- Rethink how .ext formats (v.s. ?_format=) works before 1.0 48
- Upgrade to CodeMirror 6, add SQL autocomplete 48
- JavaScript plugin hooks mechanism similar to pluggy 47
- Updated Dockerfile with SpatiaLite version 5.0 45
- Complete refactor of TableView and table.html template 45
- Port Datasette to ASGI 42
- Authentication (and permissions) as a core concept 40
- invoke_startup() is not run in some conditions, e.g. gunicorn/uvicorn workers, breaking lots of things 36
- Deploy a live instance of demos/apache-proxy 34
- await datasette.client.get(path) mechanism for executing internal requests 33
- Maintain an in-memory SQLite table of connected databases and their tables 32
- Research: demonstrate if parallel SQL queries are worthwhile 32
- Ability to sort (and paginate) by column 31
- Server hang on parallel execution of queries to named in-memory databases 31
- Default API token authentication mechanism 30
- Port as many tests as possible to async def tests against ds_client 29
- link_or_copy_directory() error - Invalid cross-device link 28
- Add ?_extra= mechanism for requesting extra properties in JSON 27
- Export to CSV 27
- base_url configuration setting 27
- Documentation with recommendations on running Datasette in production without using Docker 27
- Optimize all those calls to index_list and foreign_key_list 27
- Support cross-database joins 26
- Ability for a canned query to write to the database 26
- table.transform() method for advanced alter table 26
- New pattern for views that return either JSON or HTML, available for plugins 26
- Proof of concept for Datasette on AWS Lambda with EFS 25
- …
id | html_url ▼ | issue_url | node_id | user | created_at | updated_at | author_association | body | reactions | issue | performed_via_github_app |
---|---|---|---|---|---|---|---|---|---|---|---|
1221621700 | https://github.com/dogsheep/pocket-to-sqlite/pull/7#issuecomment-1221621700 | https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/7 | IC_kwDODLZ_YM5I0HfE | simonw 9599 | 2022-08-21T21:11:12Z | 2022-08-21T21:11:12Z | MEMBER | I thought this might need a documentation update but |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Fixed conflicting CLI flags 750141615 | |
727692413 | https://github.com/dogsheep/swarm-to-sqlite/issues/11#issuecomment-727692413 | https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/11 | MDEyOklzc3VlQ29tbWVudDcyNzY5MjQxMw== | simonw 9599 | 2020-11-16T02:15:22Z | 2020-11-16T02:15:22Z | MEMBER | Thanks, I'll look into this. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Error thrown: sqlite3.OperationalError: table users has no column named lastName 743400216 | |
761967094 | https://github.com/dogsheep/swarm-to-sqlite/issues/11#issuecomment-761967094 | https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/11 | MDEyOklzc3VlQ29tbWVudDc2MTk2NzA5NA== | simonw 9599 | 2021-01-18T04:11:13Z | 2021-01-18T04:11:13Z | MEMBER | I just got a similar error: ``` File "/home/dogsheep/datasette-venv/lib/python3.8/site-packages/swarm_to_sqlite/utils.py", line 79, in save_checkin checkins_table.m2m("users", user, m2m_table="with", pk="id") File "/home/dogsheep/datasette-venv/lib/python3.8/site-packages/sqlite_utils/db.py", line 2048, in m2m id = other_table.insert(record, pk=pk, replace=True).last_pk File "/home/dogsheep/datasette-venv/lib/python3.8/site-packages/sqlite_utils/db.py", line 1781, in insert return self.insert_all( File "/home/dogsheep/datasette-venv/lib/python3.8/site-packages/sqlite_utils/db.py", line 1899, in insert_all self.insert_chunk( File "/home/dogsheep/datasette-venv/lib/python3.8/site-packages/sqlite_utils/db.py", line 1709, in insert_chunk result = self.db.execute(query, params) File "/home/dogsheep/datasette-venv/lib/python3.8/site-packages/sqlite_utils/db.py", line 226, in execute return self.conn.execute(sql, parameters) pysqlite3.dbapi2.OperationalError: table users has no column named countryCode ``` |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Error thrown: sqlite3.OperationalError: table users has no column named lastName 743400216 | |
941274088 | https://github.com/dogsheep/swarm-to-sqlite/issues/12#issuecomment-941274088 | https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/12 | IC_kwDODD6af844GrPo | fs111 33631 | 2021-10-12T18:31:57Z | 2021-10-12T18:31:57Z | NONE | I am running into the same problem. Is there any workaround? |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
403 when getting token 951817328 | |
1502543165 | https://github.com/dogsheep/swarm-to-sqlite/issues/13#issuecomment-1502543165 | https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/13 | IC_kwDODD6af85Zjv09 | simonw 9599 | 2023-04-11T01:10:36Z | 2023-04-11T01:11:47Z | MEMBER | I just had that error myself on macOS while running the tests:
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
fails before generating views. ERR: table sqlite_master may not be modified 1373210675 | |
1502546045 | https://github.com/dogsheep/swarm-to-sqlite/issues/13#issuecomment-1502546045 | https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/13 | IC_kwDODD6af85Zjwh9 | simonw 9599 | 2023-04-11T01:14:50Z | 2023-04-11T01:14:50Z | MEMBER | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
fails before generating views. ERR: table sqlite_master may not be modified 1373210675 | ||
1502629219 | https://github.com/dogsheep/swarm-to-sqlite/issues/13#issuecomment-1502629219 | https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/13 | IC_kwDODD6af85ZkE1j | simonw 9599 | 2023-04-11T03:15:26Z | 2023-04-11T03:15:26Z | MEMBER | OK, I figured this out. Unfortunately it's an error that occurs on Python versions that have defensive mode turned on, and it doesn't look like there's a way to turn that mode off. See notes above. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
fails before generating views. ERR: table sqlite_master may not be modified 1373210675 | |
1502629404 | https://github.com/dogsheep/swarm-to-sqlite/issues/13#issuecomment-1502629404 | https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/13 | IC_kwDODD6af85ZkE4c | simonw 9599 | 2023-04-11T03:15:47Z | 2023-04-11T03:46:17Z | MEMBER | I think |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
fails before generating views. ERR: table sqlite_master may not be modified 1373210675 | |
526701674 | https://github.com/dogsheep/swarm-to-sqlite/issues/2#issuecomment-526701674 | https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/2 | MDEyOklzc3VlQ29tbWVudDUyNjcwMTY3NA== | simonw 9599 | 2019-08-30T18:24:26Z | 2019-08-30T18:24:26Z | MEMBER | I renamed |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
--save option to dump checkins to a JSON file on disk 487598468 | |
527200332 | https://github.com/dogsheep/swarm-to-sqlite/issues/3#issuecomment-527200332 | https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/3 | MDEyOklzc3VlQ29tbWVudDUyNzIwMDMzMg== | simonw 9599 | 2019-09-02T16:32:20Z | 2019-09-02T16:32:39Z | MEMBER | Also needed: an option for "fetch all checkins created within the last X days". This should help provide support for that Swarm feature where you can retroactively checkin to places in the past. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Option to fetch only checkins more recent than the current max checkin 487600595 | |
542875885 | https://github.com/dogsheep/swarm-to-sqlite/issues/3#issuecomment-542875885 | https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/3 | MDEyOklzc3VlQ29tbWVudDU0Mjg3NTg4NQ== | simonw 9599 | 2019-10-16T20:23:08Z | 2019-10-16T20:23:08Z | MEMBER | https://developer.foursquare.com/docs/api/users/checkins documents
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Option to fetch only checkins more recent than the current max checkin 487600595 | |
542876047 | https://github.com/dogsheep/swarm-to-sqlite/issues/3#issuecomment-542876047 | https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/3 | MDEyOklzc3VlQ29tbWVudDU0Mjg3NjA0Nw== | simonw 9599 | 2019-10-16T20:23:36Z | 2019-10-16T20:23:36Z | MEMBER | I'm going to go with |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Option to fetch only checkins more recent than the current max checkin 487600595 | |
542882604 | https://github.com/dogsheep/swarm-to-sqlite/issues/3#issuecomment-542882604 | https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/3 | MDEyOklzc3VlQ29tbWVudDU0Mjg4MjYwNA== | simonw 9599 | 2019-10-16T20:41:23Z | 2019-10-16T20:41:23Z | MEMBER | Documented here: https://github.com/dogsheep/swarm-to-sqlite/blob/0.2/README.md#usage |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Option to fetch only checkins more recent than the current max checkin 487600595 | |
526853542 | https://github.com/dogsheep/swarm-to-sqlite/issues/4#issuecomment-526853542 | https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/4 | MDEyOklzc3VlQ29tbWVudDUyNjg1MzU0Mg== | simonw 9599 | 2019-08-31T18:06:32Z | 2019-08-31T18:06:32Z | MEMBER | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Online tool for getting a Foursquare OAuth token 487601121 | ||
669241341 | https://github.com/dogsheep/swarm-to-sqlite/issues/8#issuecomment-669241341 | https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/8 | MDEyOklzc3VlQ29tbWVudDY2OTI0MTM0MQ== | simonw 9599 | 2020-08-05T14:55:14Z | 2020-08-05T14:55:14Z | MEMBER | Looks like there's a column that's not consistently there, so the table got created without it. Easiest fix is to add That will cause |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Error thrown: table photos has no column named hasSticker 648245071 | |
707332912 | https://github.com/dogsheep/swarm-to-sqlite/issues/8#issuecomment-707332912 | https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/8 | MDEyOklzc3VlQ29tbWVudDcwNzMzMjkxMg== | simonw 9599 | 2020-10-12T20:35:06Z | 2020-10-12T20:35:06Z | MEMBER | Shipped a fix for this in swarm-to-sqlite 0.3.2. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Error thrown: table photos has no column named hasSticker 648245071 | |
707326192 | https://github.com/dogsheep/swarm-to-sqlite/pull/10#issuecomment-707326192 | https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/10 | MDEyOklzc3VlQ29tbWVudDcwNzMyNjE5Mg== | mattiaborsoi 29426418 | 2020-10-12T20:20:02Z | 2020-10-12T20:20:02Z | CONTRIBUTOR | This closes issue #8 |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Update utils.py to fix sqlite3.OperationalError 719637258 | |
605382373 | https://github.com/dogsheep/swarm-to-sqlite/pull/6#issuecomment-605382373 | https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/6 | MDEyOklzc3VlQ29tbWVudDYwNTM4MjM3Mw== | simonw 9599 | 2020-03-28T02:27:32Z | 2020-03-28T02:27:32Z | MEMBER | Thanks! |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
don't break if source is missing 543355051 | |
541748580 | https://github.com/dogsheep/twitter-to-sqlite/issues/10#issuecomment-541748580 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/10 | MDEyOklzc3VlQ29tbWVudDU0MTc0ODU4MA== | simonw 9599 | 2019-10-14T15:30:44Z | 2019-10-14T15:30:44Z | MEMBER | Had several recommendations for https://github.com/tqdm/tqdm which is what goodreads-to-sqlite uses. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Rethink progress bars for various commands 492297930 | |
543269396 | https://github.com/dogsheep/twitter-to-sqlite/issues/10#issuecomment-543269396 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/10 | MDEyOklzc3VlQ29tbWVudDU0MzI2OTM5Ng== | simonw 9599 | 2019-10-17T17:02:07Z | 2019-10-17T17:02:07Z | MEMBER | A neat trick that Click does is detecting if an interactive terminal is attached and NOT showing a progress bar if there isn't one. Need to figure out how to do that with tqdm. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Rethink progress bars for various commands 492297930 | |
543270714 | https://github.com/dogsheep/twitter-to-sqlite/issues/10#issuecomment-543270714 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/10 | MDEyOklzc3VlQ29tbWVudDU0MzI3MDcxNA== | simonw 9599 | 2019-10-17T17:05:16Z | 2019-10-17T17:05:16Z | MEMBER | https://github.com/pallets/click/blob/716a5be90f56ce6cd506bb53d5739d09374b1636/click/_termui_impl.py#L93 is how Click does this:
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Rethink progress bars for various commands 492297930 | |
543271000 | https://github.com/dogsheep/twitter-to-sqlite/issues/10#issuecomment-543271000 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/10 | MDEyOklzc3VlQ29tbWVudDU0MzI3MTAwMA== | simonw 9599 | 2019-10-17T17:05:59Z | 2019-10-17T17:05:59Z | MEMBER | Looks like tqdm already does a TTY check here: https://github.com/tqdm/tqdm/blob/89b73bdc30c099c5b53725806e7edf3a121c9b3a/tqdm/std.py#L889-L890 |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Rethink progress bars for various commands 492297930 | |
607010791 | https://github.com/dogsheep/twitter-to-sqlite/issues/10#issuecomment-607010791 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/10 | MDEyOklzc3VlQ29tbWVudDYwNzAxMDc5MQ== | simonw 9599 | 2020-04-01T03:45:48Z | 2020-04-01T03:45:48Z | MEMBER | I'm happy with the recent work I did on this. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Rethink progress bars for various commands 492297930 | |
538711918 | https://github.com/dogsheep/twitter-to-sqlite/issues/11#issuecomment-538711918 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/11 | MDEyOklzc3VlQ29tbWVudDUzODcxMTkxOA== | simonw 9599 | 2019-10-06T04:54:17Z | 2019-10-06T04:54:17Z | MEMBER | Shipped in 0.6. Here's the documentation: https://github.com/dogsheep/twitter-to-sqlite#capturing-tweets-in-real-time-with-track-and-follow |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Commands for recording real-time tweets from the streaming API 503045221 | |
542855081 | https://github.com/dogsheep/twitter-to-sqlite/issues/12#issuecomment-542855081 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/12 | MDEyOklzc3VlQ29tbWVudDU0Mjg1NTA4MQ== | simonw 9599 | 2019-10-16T19:26:56Z | 2019-10-16T19:26:56Z | MEMBER | This may be the first case where I want to be able to repair existing databases rather than discarding their contents. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Extract "source" into a separate lookup table 503053800 | |
542855427 | https://github.com/dogsheep/twitter-to-sqlite/issues/12#issuecomment-542855427 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/12 | MDEyOklzc3VlQ29tbWVudDU0Mjg1NTQyNw== | simonw 9599 | 2019-10-16T19:27:55Z | 2019-10-16T19:27:55Z | MEMBER | I can do that by keeping |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Extract "source" into a separate lookup table 503053800 | |
542858025 | https://github.com/dogsheep/twitter-to-sqlite/issues/12#issuecomment-542858025 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/12 | MDEyOklzc3VlQ29tbWVudDU0Mjg1ODAyNQ== | simonw 9599 | 2019-10-16T19:35:31Z | 2019-10-16T19:36:09Z | MEMBER | Maybe this means I need an |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Extract "source" into a separate lookup table 503053800 | |
538804815 | https://github.com/dogsheep/twitter-to-sqlite/issues/13#issuecomment-538804815 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/13 | MDEyOklzc3VlQ29tbWVudDUzODgwNDgxNQ== | simonw 9599 | 2019-10-07T00:33:49Z | 2019-10-07T00:33:49Z | MEMBER | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
statuses-lookup command 503085013 | ||
541112108 | https://github.com/dogsheep/twitter-to-sqlite/issues/17#issuecomment-541112108 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/17 | MDEyOklzc3VlQ29tbWVudDU0MTExMjEwOA== | simonw 9599 | 2019-10-11T15:30:15Z | 2019-10-11T15:30:15Z | MEMBER | It should delete the tables entirely. That way it will work even if the table schema has changed. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
import command should empty all archive-* tables first 505674949 | |
541112588 | https://github.com/dogsheep/twitter-to-sqlite/issues/17#issuecomment-541112588 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/17 | MDEyOklzc3VlQ29tbWVudDU0MTExMjU4OA== | simonw 9599 | 2019-10-11T15:31:30Z | 2019-10-11T15:31:30Z | MEMBER | No need for an option:
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
import command should empty all archive-* tables first 505674949 | |
541118773 | https://github.com/dogsheep/twitter-to-sqlite/issues/18#issuecomment-541118773 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/18 | MDEyOklzc3VlQ29tbWVudDU0MTExODc3Mw== | simonw 9599 | 2019-10-11T15:48:31Z | 2019-10-11T15:48:31Z | MEMBER | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Command to import home-timeline 505928530 | ||
541118934 | https://github.com/dogsheep/twitter-to-sqlite/issues/18#issuecomment-541118934 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/18 | MDEyOklzc3VlQ29tbWVudDU0MTExODkzNA== | simonw 9599 | 2019-10-11T15:48:54Z | 2019-10-11T15:48:54Z | MEMBER | Rate limit is tight: 15 requests every 15 mins! |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Command to import home-timeline 505928530 | |
541119834 | https://github.com/dogsheep/twitter-to-sqlite/issues/18#issuecomment-541119834 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/18 | MDEyOklzc3VlQ29tbWVudDU0MTExOTgzNA== | simonw 9599 | 2019-10-11T15:51:22Z | 2019-10-11T16:51:33Z | MEMBER | In order to support multiple user timelines being saved in the same database, I'm going to import the tweets into the |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Command to import home-timeline 505928530 | |
541141169 | https://github.com/dogsheep/twitter-to-sqlite/issues/18#issuecomment-541141169 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/18 | MDEyOklzc3VlQ29tbWVudDU0MTE0MTE2OQ== | simonw 9599 | 2019-10-11T16:51:29Z | 2019-10-11T16:51:29Z | MEMBER | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Command to import home-timeline 505928530 | ||
541248629 | https://github.com/dogsheep/twitter-to-sqlite/issues/19#issuecomment-541248629 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/19 | MDEyOklzc3VlQ29tbWVudDU0MTI0ODYyOQ== | simonw 9599 | 2019-10-11T22:48:56Z | 2019-10-11T22:48:56Z | MEMBER |
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
since_id support for home-timeline 506087267 | |
542832952 | https://github.com/dogsheep/twitter-to-sqlite/issues/19#issuecomment-542832952 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/19 | MDEyOklzc3VlQ29tbWVudDU0MjgzMjk1Mg== | simonw 9599 | 2019-10-16T18:30:11Z | 2019-10-16T18:30:11Z | MEMBER | The
The |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
since_id support for home-timeline 506087267 | |
542849963 | https://github.com/dogsheep/twitter-to-sqlite/issues/19#issuecomment-542849963 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/19 | MDEyOklzc3VlQ29tbWVudDU0Mjg0OTk2Mw== | simonw 9599 | 2019-10-16T19:13:06Z | 2019-10-16T19:13:06Z | MEMBER | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
since_id support for home-timeline 506087267 | ||
527954898 | https://github.com/dogsheep/twitter-to-sqlite/issues/2#issuecomment-527954898 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/2 | MDEyOklzc3VlQ29tbWVudDUyNzk1NDg5OA== | simonw 9599 | 2019-09-04T15:31:46Z | 2019-09-04T15:31:46Z | MEMBER | I'm going to call this |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
"twitter-to-sqlite user-timeline" command for pulling tweets by a specific user 488833698 | |
527955302 | https://github.com/dogsheep/twitter-to-sqlite/issues/2#issuecomment-527955302 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/2 | MDEyOklzc3VlQ29tbWVudDUyNzk1NTMwMg== | simonw 9599 | 2019-09-04T15:32:39Z | 2019-09-04T15:32:39Z | MEMBER | Rate limit is 900 / 15 minutes which is 1 call per second. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
"twitter-to-sqlite user-timeline" command for pulling tweets by a specific user 488833698 | |
527990908 | https://github.com/dogsheep/twitter-to-sqlite/issues/2#issuecomment-527990908 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/2 | MDEyOklzc3VlQ29tbWVudDUyNzk5MDkwOA== | simonw 9599 | 2019-09-04T16:57:24Z | 2019-09-04T16:57:24Z | MEMBER | I just tried this using |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
"twitter-to-sqlite user-timeline" command for pulling tweets by a specific user 488833698 | |
541388038 | https://github.com/dogsheep/twitter-to-sqlite/issues/20#issuecomment-541388038 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/20 | MDEyOklzc3VlQ29tbWVudDU0MTM4ODAzOA== | simonw 9599 | 2019-10-13T05:31:58Z | 2019-10-13T05:31:58Z | MEMBER | For favourites a |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
--since support for various commands for refresh-by-cron 506268945 | |
542854749 | https://github.com/dogsheep/twitter-to-sqlite/issues/20#issuecomment-542854749 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/20 | MDEyOklzc3VlQ29tbWVudDU0Mjg1NDc0OQ== | simonw 9599 | 2019-10-16T19:26:01Z | 2019-10-16T19:26:01Z | MEMBER | I'm not going to do this for "accounts that have followed me" and "new accounts that I have followed" - instead I will recommend running the |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
--since support for various commands for refresh-by-cron 506268945 | |
544335363 | https://github.com/dogsheep/twitter-to-sqlite/issues/20#issuecomment-544335363 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/20 | MDEyOklzc3VlQ29tbWVudDU0NDMzNTM2Mw== | simonw 9599 | 2019-10-21T03:32:04Z | 2019-10-21T03:32:04Z | MEMBER | In case anyone is interested, here's an extract from the crontab I'm running these under at the moment:
|
{ "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
--since support for various commands for refresh-by-cron 506268945 | |
542333836 | https://github.com/dogsheep/twitter-to-sqlite/issues/21#issuecomment-542333836 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/21 | MDEyOklzc3VlQ29tbWVudDU0MjMzMzgzNg== | simonw 9599 | 2019-10-15T18:00:48Z | 2019-10-15T18:00:48Z | MEMBER | I'll use |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Fix & escapes in tweet text 506432572 | |
543217890 | https://github.com/dogsheep/twitter-to-sqlite/issues/23#issuecomment-543217890 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/23 | MDEyOklzc3VlQ29tbWVudDU0MzIxNzg5MA== | simonw 9599 | 2019-10-17T15:03:10Z | 2019-10-17T15:03:10Z | MEMBER | Thinking about this further: the concept of migrations may end up being in direct conflict with the I'm going to forge ahead anyway and build this because I think it will be an interesting exploration, but it's very likely this turns out to be a bad idea in the long run! |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Extremely simple migration system 508190730 | |
543222239 | https://github.com/dogsheep/twitter-to-sqlite/issues/23#issuecomment-543222239 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/23 | MDEyOklzc3VlQ29tbWVudDU0MzIyMjIzOQ== | simonw 9599 | 2019-10-17T15:12:33Z | 2019-10-17T15:12:33Z | MEMBER | Migrations will run only if you open a database that previously existed (as opposed to opening a brand new empty database). This means that the first time you run a command against a fresh database, migrations will not run and the This also means that each migration needs to be able to sanity check the database to see if it should run or not. If it should NOT run, it will do nothing but still be marked as having executed by adding to the |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Extremely simple migration system 508190730 | |
543265058 | https://github.com/dogsheep/twitter-to-sqlite/issues/25#issuecomment-543265058 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/25 | MDEyOklzc3VlQ29tbWVudDU0MzI2NTA1OA== | simonw 9599 | 2019-10-17T16:51:12Z | 2019-10-17T16:51:12Z | MEMBER | This migration function only runs if there is a table called I think this can happen if the database has just been freshly created (by a command that fetches the user's user timeline for example) and is then run a SECOND time. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Ensure migrations don't accidentally create foreign key twice 508578780 | |
543266947 | https://github.com/dogsheep/twitter-to-sqlite/issues/25#issuecomment-543266947 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/25 | MDEyOklzc3VlQ29tbWVudDU0MzI2Njk0Nw== | simonw 9599 | 2019-10-17T16:56:06Z | 2019-10-17T16:56:06Z | MEMBER | I wrote a test that proves that this is a problem. Should be an easy fix though. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Ensure migrations don't accidentally create foreign key twice 508578780 | |
547713287 | https://github.com/dogsheep/twitter-to-sqlite/issues/26#issuecomment-547713287 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/26 | MDEyOklzc3VlQ29tbWVudDU0NzcxMzI4Nw== | simonw 9599 | 2019-10-30T02:36:13Z | 2019-10-30T02:36:13Z | MEMBER | Shipped this in 0.13: https://github.com/dogsheep/twitter-to-sqlite/releases/tag/0.13 See also this Twitter thread: https://twitter.com/simonw/status/1189369677509623809 |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Command for importing mentions timeline 513074501 | |
549095217 | https://github.com/dogsheep/twitter-to-sqlite/issues/27#issuecomment-549095217 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/27 | MDEyOklzc3VlQ29tbWVudDU0OTA5NTIxNw== | simonw 9599 | 2019-11-03T01:06:25Z | 2019-11-03T01:06:25Z | MEMBER | Wow, that It looks like this needs to be combined with this API - https://developer.twitter.com/en/docs/tweets/post-and-engage/api-reference/get-statuses-retweets-id - to fetch the details of up to 100 recent users who actually DID retweet an individual status. But that has a one-every-12-seconds rate limit on it. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
retweets-of-me command 514459062 | |
549095317 | https://github.com/dogsheep/twitter-to-sqlite/issues/27#issuecomment-549095317 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/27 | MDEyOklzc3VlQ29tbWVudDU0OTA5NTMxNw== | simonw 9599 | 2019-11-03T01:08:10Z | 2019-11-03T01:08:10Z | MEMBER | Hmm... one thing that could be useful is that I'm not sure if the |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
retweets-of-me command 514459062 | |
549095463 | https://github.com/dogsheep/twitter-to-sqlite/issues/27#issuecomment-549095463 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/27 | MDEyOklzc3VlQ29tbWVudDU0OTA5NTQ2Mw== | simonw 9599 | 2019-11-03T01:10:52Z | 2019-11-03T01:10:52Z | MEMBER | I imagine it won't, since the data I would be recording and then passing to |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
retweets-of-me command 514459062 | |
549095641 | https://github.com/dogsheep/twitter-to-sqlite/issues/27#issuecomment-549095641 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/27 | MDEyOklzc3VlQ29tbWVudDU0OTA5NTY0MQ== | simonw 9599 | 2019-11-03T01:12:58Z | 2019-11-03T01:12:58Z | MEMBER | It looks like Twitter really want you to subscribe to a premium API for this kind of thing and consume retweets via webhooks: https://developer.twitter.com/en/docs/accounts-and-users/subscribe-account-activity/api-reference I'm going to give up on this for the moment. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
retweets-of-me command 514459062 | |
552133488 | https://github.com/dogsheep/twitter-to-sqlite/issues/28#issuecomment-552133488 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/28 | MDEyOklzc3VlQ29tbWVudDU1MjEzMzQ4OA== | simonw 9599 | 2019-11-09T20:15:42Z | 2019-11-09T20:15:42Z | MEMBER | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Add indexes to followers table 515658861 | ||
552129686 | https://github.com/dogsheep/twitter-to-sqlite/issues/29#issuecomment-552129686 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/29 | MDEyOklzc3VlQ29tbWVudDU1MjEyOTY4Ng== | simonw 9599 | 2019-11-09T19:27:39Z | 2019-11-09T19:27:39Z | MEMBER | I think this is fixed by the latest version of |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
`import` command fails on empty files 518725064 | |
552129921 | https://github.com/dogsheep/twitter-to-sqlite/issues/29#issuecomment-552129921 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/29 | MDEyOklzc3VlQ29tbWVudDU1MjEyOTkyMQ== | simonw 9599 | 2019-11-09T19:30:42Z | 2019-11-09T19:30:42Z | MEMBER | Confirmed, that seems to fix it:
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
`import` command fails on empty files 518725064 | |
552133449 | https://github.com/dogsheep/twitter-to-sqlite/issues/29#issuecomment-552133449 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/29 | MDEyOklzc3VlQ29tbWVudDU1MjEzMzQ0OQ== | simonw 9599 | 2019-11-09T20:15:15Z | 2019-11-09T20:15:15Z | MEMBER | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
`import` command fails on empty files 518725064 | ||
552134876 | https://github.com/dogsheep/twitter-to-sqlite/issues/29#issuecomment-552134876 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/29 | MDEyOklzc3VlQ29tbWVudDU1MjEzNDg3Ng== | jacobian 21148 | 2019-11-09T20:33:38Z | 2019-11-09T20:33:38Z | CONTRIBUTOR | ❤️ thanks! |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
`import` command fails on empty files 518725064 | |
541493242 | https://github.com/dogsheep/twitter-to-sqlite/issues/3#issuecomment-541493242 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/3 | MDEyOklzc3VlQ29tbWVudDU0MTQ5MzI0Mg== | simonw 9599 | 2019-10-14T03:35:36Z | 2019-10-14T03:35:36Z | MEMBER | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Command for running a search and saving tweets for that search 488833975 | ||
543273540 | https://github.com/dogsheep/twitter-to-sqlite/issues/3#issuecomment-543273540 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/3 | MDEyOklzc3VlQ29tbWVudDU0MzI3MzU0MA== | simonw 9599 | 2019-10-17T17:12:51Z | 2019-10-17T17:12:51Z | MEMBER | Just importing tweets here isn't enough - how are we supposed to know which tweets were imported by which search? So I think the right thing to do here is to also create a |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Command for running a search and saving tweets for that search 488833975 | |
543290744 | https://github.com/dogsheep/twitter-to-sqlite/issues/3#issuecomment-543290744 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/3 | MDEyOklzc3VlQ29tbWVudDU0MzI5MDc0NA== | simonw 9599 | 2019-10-17T17:57:14Z | 2019-10-17T17:57:14Z | MEMBER | I have a working command now. I'm going to ship it early because it could do with some other people trying it out. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Command for running a search and saving tweets for that search 488833975 | |
549096321 | https://github.com/dogsheep/twitter-to-sqlite/issues/3#issuecomment-549096321 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/3 | MDEyOklzc3VlQ29tbWVudDU0OTA5NjMyMQ== | simonw 9599 | 2019-11-03T01:27:55Z | 2019-11-03T01:28:17Z | MEMBER | It would be neat if this could support |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Command for running a search and saving tweets for that search 488833975 | |
549226399 | https://github.com/dogsheep/twitter-to-sqlite/issues/3#issuecomment-549226399 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/3 | MDEyOklzc3VlQ29tbWVudDU0OTIyNjM5OQ== | simonw 9599 | 2019-11-04T05:11:57Z | 2019-11-04T05:11:57Z | MEMBER | I'm going to add a |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Command for running a search and saving tweets for that search 488833975 | |
549228535 | https://github.com/dogsheep/twitter-to-sqlite/issues/3#issuecomment-549228535 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/3 | MDEyOklzc3VlQ29tbWVudDU0OTIyODUzNQ== | simonw 9599 | 2019-11-04T05:31:55Z | 2019-11-04T05:31:55Z | MEMBER | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Command for running a search and saving tweets for that search 488833975 | ||
552131798 | https://github.com/dogsheep/twitter-to-sqlite/issues/30#issuecomment-552131798 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/30 | MDEyOklzc3VlQ29tbWVudDU1MjEzMTc5OA== | simonw 9599 | 2019-11-09T19:54:45Z | 2019-11-09T19:54:45Z | MEMBER | Good catch - not sure how that bug crept in. Removing line 116 looks like the right fix to me. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
`followers` fails because `transform_user` is called twice 518739697 | |
552133468 | https://github.com/dogsheep/twitter-to-sqlite/issues/30#issuecomment-552133468 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/30 | MDEyOklzc3VlQ29tbWVudDU1MjEzMzQ2OA== | simonw 9599 | 2019-11-09T20:15:27Z | 2019-11-09T20:15:27Z | MEMBER | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
`followers` fails because `transform_user` is called twice 518739697 | ||
1251845216 | https://github.com/dogsheep/twitter-to-sqlite/issues/31#issuecomment-1251845216 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/31 | IC_kwDODEm0Qs5KnaRg | dckc 150986 | 2022-09-20T05:05:03Z | 2022-09-20T05:05:03Z | NONE | yay! Thanks a bunch for the The twitter "Download an archive of your data" feature doesn't include who I follow, so this is particularly handy. The whole Dogsheep thing is great :) I've written about similar things under cloud-services: - 2021: Closet Librarian Approach to Cloud Services - 2015: jukekb - Browse iTunes libraries and upload playlists to Google Music |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
"friends" command (similar to "followers") 520508502 | |
552135263 | https://github.com/dogsheep/twitter-to-sqlite/issues/31#issuecomment-552135263 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/31 | MDEyOklzc3VlQ29tbWVudDU1MjEzNTI2Mw== | simonw 9599 | 2019-11-09T20:38:35Z | 2019-11-09T20:38:35Z | MEMBER | Command still needs documentation and a bit more testing. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
"friends" command (similar to "followers") 520508502 | |
601861908 | https://github.com/dogsheep/twitter-to-sqlite/issues/34#issuecomment-601861908 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/34 | MDEyOklzc3VlQ29tbWVudDYwMTg2MTkwOA== | simonw 9599 | 2020-03-20T18:56:44Z | 2020-03-20T18:56:44Z | MEMBER | Could this be a bug in |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
IndexError running user-timeline command 585266763 | |
601871579 | https://github.com/dogsheep/twitter-to-sqlite/issues/34#issuecomment-601871579 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/34 | MDEyOklzc3VlQ29tbWVudDYwMTg3MTU3OQ== | simonw 9599 | 2020-03-20T19:20:52Z | 2020-03-20T19:20:52Z | MEMBER | Cause was the redefining of how |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
IndexError running user-timeline command 585266763 | |
601875405 | https://github.com/dogsheep/twitter-to-sqlite/issues/35#issuecomment-601875405 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/35 | MDEyOklzc3VlQ29tbWVudDYwMTg3NTQwNQ== | simonw 9599 | 2020-03-20T19:30:09Z | 2020-03-20T19:32:33Z | MEMBER | I'm going to redesign the command a little. Right now it defaults to fetching the authenticated user's timeline and allows I think the design of
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
twitter-to-sqlite user-timeline [screen_names] --sql / --attach 585282212 | |
601875524 | https://github.com/dogsheep/twitter-to-sqlite/issues/35#issuecomment-601875524 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/35 | MDEyOklzc3VlQ29tbWVudDYwMTg3NTUyNA== | simonw 9599 | 2020-03-20T19:30:27Z | 2020-03-20T19:30:27Z | MEMBER | I can give it a snazzier progress bar to, as requested by #10. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
twitter-to-sqlite user-timeline [screen_names] --sql / --attach 585282212 | |
601876051 | https://github.com/dogsheep/twitter-to-sqlite/issues/35#issuecomment-601876051 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/35 | MDEyOklzc3VlQ29tbWVudDYwMTg3NjA1MQ== | simonw 9599 | 2020-03-20T19:31:43Z | 2020-03-20T19:31:43Z | MEMBER | I'm going to keep |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
twitter-to-sqlite user-timeline [screen_names] --sql / --attach 585282212 | |
601890795 | https://github.com/dogsheep/twitter-to-sqlite/issues/35#issuecomment-601890795 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/35 | MDEyOklzc3VlQ29tbWVudDYwMTg5MDc5NQ== | simonw 9599 | 2020-03-20T20:13:08Z | 2020-03-20T20:13:08Z | MEMBER | Needs documentation. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
twitter-to-sqlite user-timeline [screen_names] --sql / --attach 585282212 | |
601892338 | https://github.com/dogsheep/twitter-to-sqlite/issues/35#issuecomment-601892338 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/35 | MDEyOklzc3VlQ29tbWVudDYwMTg5MjMzOA== | simonw 9599 | 2020-03-20T20:17:00Z | 2020-03-20T20:17:00Z | MEMBER | Released as 0.18. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
twitter-to-sqlite user-timeline [screen_names] --sql / --attach 585282212 | |
615509578 | https://github.com/dogsheep/twitter-to-sqlite/issues/37#issuecomment-615509578 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/37 | MDEyOklzc3VlQ29tbWVudDYxNTUwOTU3OA== | simonw 9599 | 2020-04-17T23:36:00Z | 2020-04-17T23:36:00Z | MEMBER |
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Handle "User not found" error 585353598 | |
615509803 | https://github.com/dogsheep/twitter-to-sqlite/issues/37#issuecomment-615509803 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/37 | MDEyOklzc3VlQ29tbWVudDYxNTUwOTgwMw== | simonw 9599 | 2020-04-17T23:36:40Z | 2020-04-17T23:36:40Z | MEMBER | I'm going to print a warning to stderr, skip and continue - because if you have 100 screen names and only one of them is invalid you should still execute for the other 99. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Handle "User not found" error 585353598 | |
615510361 | https://github.com/dogsheep/twitter-to-sqlite/issues/37#issuecomment-615510361 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/37 | MDEyOklzc3VlQ29tbWVudDYxNTUxMDM2MQ== | simonw 9599 | 2020-04-17T23:38:27Z | 2020-04-17T23:38:27Z | MEMBER | That's a bit tricky since I'd have to rewrite the internals of a bunch of other commands. For the moment I'll exit the script with an error but at least it will be a decent error! |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Handle "User not found" error 585353598 | |
601939656 | https://github.com/dogsheep/twitter-to-sqlite/issues/38#issuecomment-601939656 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/38 | MDEyOklzc3VlQ29tbWVudDYwMTkzOTY1Ng== | simonw 9599 | 2020-03-20T22:37:04Z | 2020-03-20T22:37:04Z | MEMBER | New formatting:
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Screen name display for user-timeline is uneven 585359363 | |
606304837 | https://github.com/dogsheep/twitter-to-sqlite/issues/39#issuecomment-606304837 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/39 | MDEyOklzc3VlQ29tbWVudDYwNjMwNDgzNw== | simonw 9599 | 2020-03-30T23:27:50Z | 2020-03-30T23:29:31Z | MEMBER | One option would be something like this:
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
--since feature can be confused by retweets 590666760 | |
606305701 | https://github.com/dogsheep/twitter-to-sqlite/issues/39#issuecomment-606305701 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/39 | MDEyOklzc3VlQ29tbWVudDYwNjMwNTcwMQ== | simonw 9599 | 2020-03-30T23:30:27Z | 2020-03-30T23:30:27Z | MEMBER | A better alternative would be to maintain a separate table with the last seen since value for when we ran |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
--since feature can be confused by retweets 590666760 | |
606309165 | https://github.com/dogsheep/twitter-to-sqlite/issues/39#issuecomment-606309165 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/39 | MDEyOklzc3VlQ29tbWVudDYwNjMwOTE2NQ== | simonw 9599 | 2020-03-30T23:41:31Z | 2020-03-30T23:41:31Z | MEMBER | I like the separate |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
--since feature can be confused by retweets 590666760 | |
606824992 | https://github.com/dogsheep/twitter-to-sqlite/issues/39#issuecomment-606824992 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/39 | MDEyOklzc3VlQ29tbWVudDYwNjgyNDk5Mg== | simonw 9599 | 2020-03-31T19:24:23Z | 2020-03-31T19:24:23Z | MEMBER | The
All of them use the same Question: should I have a table for each of those four methods or a single table that is used by them all? I'm leaning towards four separate tables. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
--since feature can be confused by retweets 590666760 | |
606843224 | https://github.com/dogsheep/twitter-to-sqlite/issues/39#issuecomment-606843224 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/39 | MDEyOklzc3VlQ29tbWVudDYwNjg0MzIyNA== | simonw 9599 | 2020-03-31T19:59:11Z | 2020-03-31T20:06:32Z | MEMBER | Or... have a single
That last example would use the hash generated here: |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
--since feature can be confused by retweets 590666760 | |
606844521 | https://github.com/dogsheep/twitter-to-sqlite/issues/39#issuecomment-606844521 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/39 | MDEyOklzc3VlQ29tbWVudDYwNjg0NDUyMQ== | simonw 9599 | 2020-03-31T20:01:39Z | 2020-03-31T20:01:39Z | MEMBER | I think |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
--since feature can be confused by retweets 590666760 | |
606850008 | https://github.com/dogsheep/twitter-to-sqlite/issues/39#issuecomment-606850008 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/39 | MDEyOklzc3VlQ29tbWVudDYwNjg1MDAwOA== | simonw 9599 | 2020-03-31T20:13:59Z | 2020-04-01T00:23:00Z | MEMBER | Table design for type | key | since_id --- | --- | --- 1 | 124324 | 2347239847293 2 | 99ff9cefff5cbfd804f7cd43e2b27ced8addbe8d | 2125947927344 Primary compound key on
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
--since feature can be confused by retweets 590666760 | |
606850453 | https://github.com/dogsheep/twitter-to-sqlite/issues/39#issuecomment-606850453 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/39 | MDEyOklzc3VlQ29tbWVudDYwNjg1MDQ1Mw== | simonw 9599 | 2020-03-31T20:14:58Z | 2020-04-01T03:03:50Z | MEMBER | Actually I'll hard-code the population of |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
--since feature can be confused by retweets 590666760 | |
606998669 | https://github.com/dogsheep/twitter-to-sqlite/issues/39#issuecomment-606998669 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/39 | MDEyOklzc3VlQ29tbWVudDYwNjk5ODY2OQ== | simonw 9599 | 2020-04-01T02:57:36Z | 2020-04-01T02:57:36Z | MEMBER | The tricky thing here is thinking about the interaction between the recorded since_id and a desire to run the initial import. The first time you run We need to record the maximum ID from those as the But what happens if our initial import is cancelled after only a few tweets? We risk never pulling in the rest of the tweets. Not sure if I need to solve this at all or if I should instead trust users to run the command a second time without I had considered letting |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
--since feature can be confused by retweets 590666760 | |
607003655 | https://github.com/dogsheep/twitter-to-sqlite/issues/39#issuecomment-607003655 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/39 | MDEyOklzc3VlQ29tbWVudDYwNzAwMzY1NQ== | simonw 9599 | 2020-04-01T03:18:00Z | 2020-04-01T03:18:00Z | MEMBER | I've got this working for the |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
--since feature can be confused by retweets 590666760 | |
607010634 | https://github.com/dogsheep/twitter-to-sqlite/issues/39#issuecomment-607010634 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/39 | MDEyOklzc3VlQ29tbWVudDYwNzAxMDYzNA== | simonw 9599 | 2020-04-01T03:45:16Z | 2020-04-01T03:45:16Z | MEMBER | OK, fix is applied to everything now. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
--since feature can be confused by retweets 590666760 | |
527682713 | https://github.com/dogsheep/twitter-to-sqlite/issues/4#issuecomment-527682713 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/4 | MDEyOklzc3VlQ29tbWVudDUyNzY4MjcxMw== | simonw 9599 | 2019-09-03T23:48:57Z | 2019-09-03T23:48:57Z | MEMBER | One interesting challenge here is that the JSON format for tweets in the archive is subtly different from the JSON format currently returned by the API. If we want to keep the tweets in the same database table (which feels like the right thing to me) we'll need to handle this. One thing we can do is have a column for We can also ensure that tweets from the API always over-write the version that came from the archive (using |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Command for importing data from a Twitter Export file 488835586 | |
540879620 | https://github.com/dogsheep/twitter-to-sqlite/issues/4#issuecomment-540879620 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/4 | MDEyOklzc3VlQ29tbWVudDU0MDg3OTYyMA== | simonw 9599 | 2019-10-11T02:59:16Z | 2019-10-11T02:59:16Z | MEMBER | Also import ad preferences and all that other junk. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Command for importing data from a Twitter Export file 488835586 | |
606307019 | https://github.com/dogsheep/twitter-to-sqlite/issues/40#issuecomment-606307019 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/40 | MDEyOklzc3VlQ29tbWVudDYwNjMwNzAxOQ== | simonw 9599 | 2020-03-30T23:34:27Z | 2020-03-30T23:34:27Z | MEMBER | The count properties available for a user are:
May as well track history for all of them? Should be pretty cheap to store. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Feature: record history of follower counts 590669793 | |
606307376 | https://github.com/dogsheep/twitter-to-sqlite/issues/40#issuecomment-606307376 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/40 | MDEyOklzc3VlQ29tbWVudDYwNjMwNzM3Ng== | simonw 9599 | 2020-03-30T23:35:40Z | 2020-03-30T23:39:15Z | MEMBER | I think five separate tables:
Each with the following structure:
I'm tempted to have a compound primary key here - user, datetime |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Feature: record history of follower counts 590669793 | |
607011421 | https://github.com/dogsheep/twitter-to-sqlite/issues/40#issuecomment-607011421 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/40 | MDEyOklzc3VlQ29tbWVudDYwNzAxMTQyMQ== | simonw 9599 | 2020-04-01T03:47:37Z | 2020-04-01T03:55:08Z | MEMBER | Actually a single table with a In which case the compound primary key would be |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Feature: record history of follower counts 590669793 | |
607011972 | https://github.com/dogsheep/twitter-to-sqlite/issues/40#issuecomment-607011972 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/40 | MDEyOklzc3VlQ29tbWVudDYwNzAxMTk3Mg== | simonw 9599 | 2020-04-01T03:49:02Z | 2020-04-01T03:50:01Z | MEMBER | I want the datetime value to look like
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Feature: record history of follower counts 590669793 | |
607019151 | https://github.com/dogsheep/twitter-to-sqlite/issues/40#issuecomment-607019151 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/40 | MDEyOklzc3VlQ29tbWVudDYwNzAxOTE1MQ== | simonw 9599 | 2020-04-01T04:11:10Z | 2020-04-01T04:11:10Z | MEMBER | In testing this collects a LOT of data. I'm going to skip tracking favourites_count and statuses_count and just track followers, friends and listed instead. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Feature: record history of follower counts 590669793 | |
615513491 | https://github.com/dogsheep/twitter-to-sqlite/issues/43#issuecomment-615513491 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/43 | MDEyOklzc3VlQ29tbWVudDYxNTUxMzQ5MQ== | simonw 9599 | 2020-04-17T23:48:28Z | 2020-04-17T23:48:28Z | MEMBER | Released in 0.21. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
"twitter-to-sqlite lists" command for retrieving a user's owned lists 602176870 | |
616029262 | https://github.com/dogsheep/twitter-to-sqlite/issues/45#issuecomment-616029262 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/45 | MDEyOklzc3VlQ29tbWVudDYxNjAyOTI2Mg== | simonw 9599 | 2020-04-19T04:39:21Z | 2020-04-19T04:39:21Z | MEMBER | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Use raise_for_status() everywhere 602619330 | ||
645512127 | https://github.com/dogsheep/twitter-to-sqlite/issues/47#issuecomment-645512127 | https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/47 | MDEyOklzc3VlQ29tbWVudDY0NTUxMjEyNw== | simonw 9599 | 2020-06-17T17:24:22Z | 2020-06-17T17:24:22Z | MEMBER | That means your version of SQLite is old enough that it doesn't support the FTS5 extension. Could you share what operating system you're running, and what the output is that you get from running this?
I can teach this tool to fall back on FTS4 if FTS5 isn't available. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Fall back to FTS4 if FTS5 is not available 639542974 |
Advanced export
JSON shape: default, array, newline-delimited, object
CREATE TABLE [issue_comments] ( [html_url] TEXT, [issue_url] TEXT, [id] INTEGER PRIMARY KEY, [node_id] TEXT, [user] INTEGER REFERENCES [users]([id]), [created_at] TEXT, [updated_at] TEXT, [author_association] TEXT, [body] TEXT, [reactions] TEXT, [issue] INTEGER REFERENCES [issues]([id]) , [performed_via_github_app] TEXT); CREATE INDEX [idx_issue_comments_issue] ON [issue_comments] ([issue]); CREATE INDEX [idx_issue_comments_user] ON [issue_comments] ([user]);
user >30