{"html_url": "https://github.com/simonw/sqlite-utils/issues/149#issuecomment-688482355", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/149", "id": 688482355, "node_id": "MDEyOklzc3VlQ29tbWVudDY4ODQ4MjM1NQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-07T19:22:51Z", "updated_at": "2020-09-07T19:22:51Z", "author_association": "OWNER", "body": "And the SQLite documentation says:\r\n> When the REPLACE conflict resolution strategy deletes rows in order to satisfy a constraint, [delete triggers](https://www.sqlite.org/lang_createtrigger.html) fire if and only if [recursive triggers](https://www.sqlite.org/pragma.html#pragma_recursive_triggers) are enabled.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 695319258, "label": "FTS table with 7 rows has _fts_docsize table with 9,141 rows"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/149#issuecomment-688482055", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/149", "id": 688482055, "node_id": "MDEyOklzc3VlQ29tbWVudDY4ODQ4MjA1NQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-07T19:21:42Z", "updated_at": "2020-09-07T19:21:42Z", "author_association": "OWNER", "body": "Using `replace=True` there executes `INSERT OR REPLACE` - and Dan Kennedy (SQLite maintainer) on the SQLite forums said this:\r\n> Are you using \"REPLACE INTO\", or \"UPDATE OR REPLACE\" on the \"licenses\" table without having first executed \"PRAGMA recursive_triggers = 1\"? The docs note that delete triggers will not be fired in this case, which would explain things. Second paragraph under \"REPLACE\" here:\r\n>\r\n> https://www.sqlite.org/lang_conflict.html", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 695319258, "label": "FTS table with 7 rows has _fts_docsize table with 9,141 rows"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/149#issuecomment-688481374", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/149", "id": 688481374, "node_id": "MDEyOklzc3VlQ29tbWVudDY4ODQ4MTM3NA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-07T19:19:08Z", "updated_at": "2020-09-07T19:19:08Z", "author_association": "OWNER", "body": "reading through the code for `github-to-sqlite repos` - one of the things it does is calls `save_license` for each repo:\r\n\r\nhttps://github.com/dogsheep/github-to-sqlite/blob/39b2234253096bd579feed4e25104698b8ccd2ba/github_to_sqlite/utils.py#L259-L262\r\n\r\n```python\r\ndef save_license(db, license):\r\n if license is None:\r\n return None\r\n return db[\"licenses\"].insert(license, pk=\"key\", replace=True).last_pk\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 695319258, "label": "FTS table with 7 rows has _fts_docsize table with 9,141 rows"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/146#issuecomment-688481317", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/146", "id": 688481317, "node_id": "MDEyOklzc3VlQ29tbWVudDY4ODQ4MTMxNw==", "user": {"value": 96218, "label": "simonwiles"}, "created_at": "2020-09-07T19:18:55Z", "updated_at": "2020-09-07T19:18:55Z", "author_association": "CONTRIBUTOR", "body": "Just force-pushed to update d042f9c with more formatting changes to satisfy `black==20.8b1` and pass the GitHub Actions \"Test\" workflow.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 688668680, "label": "Handle case where subsequent records (after first batch) include extra columns"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/149#issuecomment-688480665", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/149", "id": 688480665, "node_id": "MDEyOklzc3VlQ29tbWVudDY4ODQ4MDY2NQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-07T19:16:20Z", "updated_at": "2020-09-07T19:16:20Z", "author_association": "OWNER", "body": "Aha! I have managed to replicate the bug:\r\n```\r\n(github-to-sqlite) /tmp % sqlite-utils tables --counts github.db | grep licenses\r\n {\"table\": \"licenses\", \"count\": 7},\r\n {\"table\": \"licenses_fts_data\", \"count\": 35},\r\n {\"table\": \"licenses_fts_idx\", \"count\": 16},\r\n {\"table\": \"licenses_fts_docsize\", \"count\": 9151},\r\n {\"table\": \"licenses_fts_config\", \"count\": 1},\r\n {\"table\": \"licenses_fts\", \"count\": 7},\r\n(github-to-sqlite) /tmp % github-to-sqlite repos github.db dogsheep \r\n(github-to-sqlite) /tmp % sqlite-utils tables --counts github.db | grep licenses\r\n {\"table\": \"licenses\", \"count\": 7},\r\n {\"table\": \"licenses_fts_data\", \"count\": 45},\r\n {\"table\": \"licenses_fts_idx\", \"count\": 26},\r\n {\"table\": \"licenses_fts_docsize\", \"count\": 9161},\r\n {\"table\": \"licenses_fts_config\", \"count\": 1},\r\n {\"table\": \"licenses_fts\", \"count\": 7},\r\n```\r\nNote that the number of records in `licenses_fts_docsize` went from 9151 to 9161.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 695319258, "label": "FTS table with 7 rows has _fts_docsize table with 9,141 rows"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/146#issuecomment-688479163", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/146", "id": 688479163, "node_id": "MDEyOklzc3VlQ29tbWVudDY4ODQ3OTE2Mw==", "user": {"value": 96218, "label": "simonwiles"}, "created_at": "2020-09-07T19:10:33Z", "updated_at": "2020-09-07T19:11:57Z", "author_association": "CONTRIBUTOR", "body": "@simonw -- I've gone ahead updated the documentation to reflect the changes introduced in this PR. IMO it's ready to merge now.\r\n\r\nIn writing the documentation changes, I begin to wonder about the value and role of `batch_size` at all, tbh. May I assume it was originally intended to prevent using the entire row set to determine columns and column types, and that this was a performance consideration? If so, this PR entirely undermines its purpose. I've been passing in excess of 500,000 rows at a time to `insert_all()` with these changes and although I'm sure the performance difference is measurable it's not really noticeable; given #145, I don't know that any performance advantages outweigh the problems doing it this way removes. What do you think about just dropping the argument and defaulting to the maximum `batch_size` permissible given `SQLITE_MAX_VARS`? Are there other reasons one might want to restrict `batch_size` that I've overlooked? I could open a new issue to discuss/implement this.\r\n\r\nOf course the documentation will need to change again too if/when something is done about #147.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 688668680, "label": "Handle case where subsequent records (after first batch) include extra columns"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/149#issuecomment-688464181", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/149", "id": 688464181, "node_id": "MDEyOklzc3VlQ29tbWVudDY4ODQ2NDE4MQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-07T18:19:54Z", "updated_at": "2020-09-07T18:19:54Z", "author_association": "OWNER", "body": "Even though that table doesn't declare an integer primary key it does have a `rowid` column: https://github-to-sqlite.dogsheep.net/github?sql=select+rowid%2C+%5Bkey%5D%2C+name%2C+spdx_id%2C+url%2C+node_id+from+licenses+order+by+%5Bkey%5D+limit+101\r\n\r\n| rowid | key | name | spdx_id | url | node_id |\r\n| --- | --- | --- | --- | --- | --- |\r\n| 9150 | apache-2.0 | Apache License 2.0 | Apache-2.0 | | MDc6TGljZW5zZTI= |\r\n| 112 | bsd-3-clause | BSD 3-Clause \"New\" or \"Revised\" License | BSD-3-Clause | | MDc6TGljZW5zZTU= |\r\n\r\nhttps://www.sqlite.org/rowidtable.html explains has this clue:\r\n\r\n> If the rowid is not aliased by INTEGER PRIMARY KEY then it is not persistent and might change. In particular the VACUUM command will change rowids for tables that do not declare an INTEGER PRIMARY KEY. Therefore, applications should not normally access the rowid directly, but instead use an INTEGER PRIMARY KEY. ", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 695319258, "label": "FTS table with 7 rows has _fts_docsize table with 9,141 rows"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/149#issuecomment-688460865", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/149", "id": 688460865, "node_id": "MDEyOklzc3VlQ29tbWVudDY4ODQ2MDg2NQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-07T18:07:14Z", "updated_at": "2020-09-07T18:07:14Z", "author_association": "OWNER", "body": "Another likely culprit: `licenses` has a text primary key, so it's not using `rowid`:\r\n```sql\r\nCREATE TABLE [licenses] (\r\n [key] TEXT PRIMARY KEY,\r\n [name] TEXT,\r\n [spdx_id] TEXT,\r\n [url] TEXT,\r\n [node_id] TEXT\r\n);\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 695319258, "label": "FTS table with 7 rows has _fts_docsize table with 9,141 rows"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/149#issuecomment-688460729", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/149", "id": 688460729, "node_id": "MDEyOklzc3VlQ29tbWVudDY4ODQ2MDcyOQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-07T18:06:44Z", "updated_at": "2020-09-07T18:06:44Z", "author_association": "OWNER", "body": "First posted on SQLite forum here but I'm pretty sure this is a bug in how `sqlite-utils` created those tables: https://sqlite.org/forum/forumpost/51aada1b45", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 695319258, "label": "FTS table with 7 rows has _fts_docsize table with 9,141 rows"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/148#issuecomment-688434226", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/148", "id": 688434226, "node_id": "MDEyOklzc3VlQ29tbWVudDY4ODQzNDIyNg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-07T16:50:33Z", "updated_at": "2020-09-07T16:50:33Z", "author_association": "OWNER", "body": "This may be as easy as applying `textwrap.dedent()` to this: https://github.com/simonw/sqlite-utils/blob/0e62744da9a429093e3409575c1f881376b0361f/sqlite_utils/db.py#L778-L787\r\n\r\nI could apply that to a few other queries in that code as well.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 695276328, "label": "More attractive indentation of created FTS table schema"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/dogsheep-beta/issues/17#issuecomment-687880459", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/17", "id": 687880459, "node_id": "MDEyOklzc3VlQ29tbWVudDY4Nzg4MDQ1OQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-06T19:36:32Z", "updated_at": "2020-09-06T19:36:32Z", "author_association": "MEMBER", "body": "At some point I may even want to support search types which are indexed from (and inflated from) more than one database file. I'm going to ignore that for the moment though.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 694500679, "label": "Rename \"table\" to \"type\""}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/dogsheep-beta/issues/13#issuecomment-686774592", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/13", "id": 686774592, "node_id": "MDEyOklzc3VlQ29tbWVudDY4Njc3NDU5Mg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-03T21:30:21Z", "updated_at": "2020-09-03T21:30:21Z", "author_association": "MEMBER", "body": "This is partially supported: the custom search SQL we run doesn't escape them, but the `?_search` used to calculate facet counts does. So this is a bug.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 692386625, "label": "Support advanced FTS queries"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/dogsheep-beta/issues/9#issuecomment-686767208", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/9", "id": 686767208, "node_id": "MDEyOklzc3VlQ29tbWVudDY4Njc2NzIwOA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-03T21:12:14Z", "updated_at": "2020-09-03T21:12:14Z", "author_association": "MEMBER", "body": "Documentation: https://github.com/dogsheep/dogsheep-beta/blob/0.4/README.md#custom-results-display", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 691521965, "label": "Mechanism for defining custom display of results"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/dogsheep-beta/issues/3#issuecomment-686689612", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/3", "id": 686689612, "node_id": "MDEyOklzc3VlQ29tbWVudDY4NjY4OTYxMg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-03T18:44:20Z", "updated_at": "2020-09-03T18:44:20Z", "author_association": "MEMBER", "body": "Facets are now displayed but selecting them doesn't work yet.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 689810340, "label": "Datasette plugin to provide custom page for running faceted, ranked searches"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/dogsheep-beta/issues/5#issuecomment-686689366", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/5", "id": 686689366, "node_id": "MDEyOklzc3VlQ29tbWVudDY4NjY4OTM2Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-03T18:43:50Z", "updated_at": "2020-09-03T18:43:50Z", "author_association": "MEMBER", "body": "No longer needed thanks to #9", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 689847361, "label": "Add a context column that's not searchable"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/dogsheep-beta/issues/9#issuecomment-686689122", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/9", "id": 686689122, "node_id": "MDEyOklzc3VlQ29tbWVudDY4NjY4OTEyMg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-03T18:43:20Z", "updated_at": "2020-09-03T18:43:20Z", "author_association": "MEMBER", "body": "Needs documentation.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 691521965, "label": "Mechanism for defining custom display of results"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/dogsheep-beta/issues/9#issuecomment-686688963", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/9", "id": 686688963, "node_id": "MDEyOklzc3VlQ29tbWVudDY4NjY4ODk2Mw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-03T18:42:59Z", "updated_at": "2020-09-03T18:42:59Z", "author_association": "MEMBER", "body": "I'm pleased with how this works now.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 691521965, "label": "Mechanism for defining custom display of results"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/dogsheep-beta/issues/11#issuecomment-686618669", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/11", "id": 686618669, "node_id": "MDEyOklzc3VlQ29tbWVudDY4NjYxODY2OQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-03T16:47:34Z", "updated_at": "2020-09-03T16:53:25Z", "author_association": "MEMBER", "body": "I think a `is_public` integer column which defaults to 0 would be good here.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 692125110, "label": "Public / Private mechanism"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/dogsheep-beta/issues/10#issuecomment-686238498", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/10", "id": 686238498, "node_id": "MDEyOklzc3VlQ29tbWVudDY4NjIzODQ5OA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-03T04:05:05Z", "updated_at": "2020-09-03T04:05:05Z", "author_association": "MEMBER", "body": "Since the first two categories are `created` and `saved` this one should be called `received`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 691557547, "label": "Category 3: received"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/dogsheep-beta/issues/9#issuecomment-686163754", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/9", "id": 686163754, "node_id": "MDEyOklzc3VlQ29tbWVudDY4NjE2Mzc1NA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-03T00:46:21Z", "updated_at": "2020-09-03T00:46:21Z", "author_association": "MEMBER", "body": "Challenge: the `dogsheep-beta.yml` configuration file that is passed to the `dogsheep-beta index` command needs to also be made available to Datasette itself, so that it can read the configuration.\r\n\r\nLet's say it can either be duplicated in the `plugins` configuration block of the `metadata.yml` OR you can do this in `metadata.yml`:\r\n\r\n```yaml\r\nplugins:\r\n dogsheep-beta:\r\n config_file: dogsheep-beta.yml\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 691521965, "label": "Mechanism for defining custom display of results"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/dogsheep-beta/issues/9#issuecomment-686158454", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/9", "id": 686158454, "node_id": "MDEyOklzc3VlQ29tbWVudDY4NjE1ODQ1NA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-03T00:32:42Z", "updated_at": "2020-09-03T00:32:42Z", "author_association": "MEMBER", "body": "If this turns out to be too inefficient I could add a `display` text column to the `search_index` table which is designed to be populated with arbitrary JSON by the indexing query, which can then be used to render the template fragment.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 691521965, "label": "Mechanism for defining custom display of results"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/dogsheep-beta/issues/9#issuecomment-686154627", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/9", "id": 686154627, "node_id": "MDEyOklzc3VlQ29tbWVudDY4NjE1NDYyNw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-03T00:19:22Z", "updated_at": "2020-09-03T00:19:22Z", "author_association": "MEMBER", "body": "If this performs well enough (100 displayed items will be 100 extra `display_sql` calls) then I'll go with this as the design for the feature.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 691521965, "label": "Mechanism for defining custom display of results"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/dogsheep-beta/issues/9#issuecomment-686154486", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/9", "id": 686154486, "node_id": "MDEyOklzc3VlQ29tbWVudDY4NjE1NDQ4Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-03T00:18:54Z", "updated_at": "2020-09-03T00:18:54Z", "author_association": "MEMBER", "body": "`display_sql` could be optional. If it's not defined, a `row` object is passed to the template which is the row that's stored in `search_index`. If `display_sql` IS defined then it's executed and the result is made available as a `display` object in addition to the `row` object.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 691521965, "label": "Mechanism for defining custom display of results"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/dogsheep-beta/issues/9#issuecomment-686153967", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/9", "id": 686153967, "node_id": "MDEyOklzc3VlQ29tbWVudDY4NjE1Mzk2Nw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-03T00:17:16Z", "updated_at": "2020-09-03T00:17:55Z", "author_association": "MEMBER", "body": "Maybe I can take advantage of https://sqlite.org/np1queryprob.html here - I could define a SQL query for fetching the \"display\" version of each item, and include a Jinja template fragment in the configuration as well. Maybe something like this:\r\n```yaml\r\nphotos.db:\r\n photos_with_apple_metadata:\r\n sql: |-\r\n select\r\n sha256 as key,\r\n 'Photo in ' || coalesce(place_city, 'unknown') as title,\r\n (\r\n select\r\n group_concat(normalized_string, ' ')\r\n from\r\n labels\r\n where\r\n labels.uuid = photos_with_apple_metadata.uuid\r\n ) as search_1,\r\n date as timestamp,\r\n 1 as category\r\n from\r\n photos_with_apple_metadata\r\n display_sql: |-\r\n select\r\n sha256, place_city, date\r\n from photos_with_apple_metadata\r\n where sha256 = :key\r\n display: |-\r\n \r\n

Taken in {{ display.place_city }} on {{ display.date }}

\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 691521965, "label": "Mechanism for defining custom display of results"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/pull/952#issuecomment-686061028", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/952", "id": 686061028, "node_id": "MDEyOklzc3VlQ29tbWVudDY4NjA2MTAyOA==", "user": {"value": 27856297, "label": "dependabot-preview[bot]"}, "created_at": "2020-09-02T22:26:14Z", "updated_at": "2020-09-02T22:26:14Z", "author_association": "CONTRIBUTOR", "body": "Looks like black is up-to-date now, so this is no longer needed.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 687245650, "label": "Update black requirement from ~=19.10b0 to >=19.10,<21.0"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/dogsheep-beta/issues/7#issuecomment-685970384", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/7", "id": 685970384, "node_id": "MDEyOklzc3VlQ29tbWVudDY4NTk3MDM4NA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-02T20:11:41Z", "updated_at": "2020-09-02T20:11:59Z", "author_association": "MEMBER", "body": "Default categories:\r\n\r\n- 1 = created\r\n- 2 = saved", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 691265198, "label": "Mechanism for differentiating between \"by me\" and \"liked by me\""}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/dogsheep-beta/issues/7#issuecomment-685966707", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/7", "id": 685966707, "node_id": "MDEyOklzc3VlQ29tbWVudDY4NTk2NjcwNw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-02T20:04:08Z", "updated_at": "2020-09-02T20:04:08Z", "author_association": "MEMBER", "body": "I'll make `category` a foreign key to a `categories` table so Datasette can automatically show the `name` column.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 691265198, "label": "Mechanism for differentiating between \"by me\" and \"liked by me\""}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/dogsheep-beta/issues/7#issuecomment-685966361", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/7", "id": 685966361, "node_id": "MDEyOklzc3VlQ29tbWVudDY4NTk2NjM2MQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-02T20:03:29Z", "updated_at": "2020-09-02T20:03:41Z", "author_association": "MEMBER", "body": "I'm going to implement the first version of this as an indexed integer `category` column which has 1 for \"about me\" and 2 for \"liked by me\" - and space for other category numerals in the future, albeit a row can only belong to one category.\r\n\r\nI'll think about a full tagging system separately.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 691265198, "label": "Mechanism for differentiating between \"by me\" and \"liked by me\""}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/dogsheep-beta/issues/7#issuecomment-685965516", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/7", "id": 685965516, "node_id": "MDEyOklzc3VlQ29tbWVudDY4NTk2NTUxNg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-02T20:01:54Z", "updated_at": "2020-09-02T20:01:54Z", "author_association": "MEMBER", "body": "Relevant post: https://sqlite.org/forum/forumpost/9f06fedaa5 - drh says:\r\n\r\n> Indexes are one-to-one. There is one entry in the index for each row in the table.\r\n>\r\n> You are asking for an index that is many-to-one - multiple index entries for each table row.\r\n>\r\n> A Full-Text Index is basically a many-to-one index. So if all of your array entries really are words, you could probably get this to work using a Full-Text Index.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 691265198, "label": "Mechanism for differentiating between \"by me\" and \"liked by me\""}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/dogsheep-beta/issues/7#issuecomment-685962280", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/7", "id": 685962280, "node_id": "MDEyOklzc3VlQ29tbWVudDY4NTk2MjI4MA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-02T19:55:26Z", "updated_at": "2020-09-02T19:59:58Z", "author_association": "MEMBER", "body": "Relevant: https://charlesleifer.com/blog/a-tour-of-tagging-schemas-many-to-many-bitmaps-and-more/\r\n\r\nSQLite supports bitwise operators Binary AND (&) and Binary OR (|) - I could try those. Not sure how they interact with indexes though.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 691265198, "label": "Mechanism for differentiating between \"by me\" and \"liked by me\""}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/dogsheep-beta/issues/3#issuecomment-685961809", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/3", "id": 685961809, "node_id": "MDEyOklzc3VlQ29tbWVudDY4NTk2MTgwOQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-02T19:54:24Z", "updated_at": "2020-09-02T19:54:24Z", "author_association": "MEMBER", "body": "This should implement search highlighting too, as seen on https://til.simonwillison.net/til/search?q=cloud\r\n\r\n\"TIL_search__cloud\"\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 689810340, "label": "Datasette plugin to provide custom page for running faceted, ranked searches"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/dogsheep-beta/issues/8#issuecomment-685960072", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/8", "id": 685960072, "node_id": "MDEyOklzc3VlQ29tbWVudDY4NTk2MDA3Mg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-02T19:50:47Z", "updated_at": "2020-09-02T19:50:47Z", "author_association": "MEMBER", "body": "This doesn't actually help, because the Datasette table view page doesn't then support adding the `where search_index_fts match :query` bit.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 691369691, "label": "Create a view for running faceted searches"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/dogsheep-beta/issues/7#issuecomment-685895540", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/7", "id": 685895540, "node_id": "MDEyOklzc3VlQ29tbWVudDY4NTg5NTU0MA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-02T17:46:44Z", "updated_at": "2020-09-02T17:46:44Z", "author_association": "MEMBER", "body": "Some opet questions about this:\r\n\r\n- Should I restrict to two exclusive categories here, or should I have a generic category mechanism that can be expanded to more than two?\r\n- Should an item be able to exist in more than one category? Do I want to be able to mark an indexed item as both by-me and liked-by-me for example? This question is more interesting if the number of categories is greater than two.\r\n- How should this be modeled? Single column, multiple boolean columns, JSON array, m2m against separate table?\r\n- What's the best way to make this performant", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 691265198, "label": "Mechanism for differentiating between \"by me\" and \"liked by me\""}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/dogsheep-beta/issues/2#issuecomment-685121074", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/2", "id": 685121074, "node_id": "MDEyOklzc3VlQ29tbWVudDY4NTEyMTA3NA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-01T20:42:00Z", "updated_at": "2020-09-01T20:42:00Z", "author_association": "MEMBER", "body": "Documentation at the bottom of the Usage section here: https://github.com/dogsheep/dogsheep-beta/blob/0.2/README.md#usage", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 689809225, "label": "Apply porter stemming"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/dogsheep-beta/issues/2#issuecomment-685115519", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/2", "id": 685115519, "node_id": "MDEyOklzc3VlQ29tbWVudDY4NTExNTUxOQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-01T20:31:57Z", "updated_at": "2020-09-01T20:31:57Z", "author_association": "MEMBER", "body": "Actually this doesn't work: you can't turn on stemming for specific tables, because all of the content goes into a single `search_index` table which is configured the same way.\r\n\r\nSo stemming needs to be a global option.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 689809225, "label": "Apply porter stemming"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/pocket-to-sqlite/issues/5#issuecomment-684425714", "issue_url": "https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/5", "id": 684425714, "node_id": "MDEyOklzc3VlQ29tbWVudDY4NDQyNTcxNA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-01T06:18:32Z", "updated_at": "2020-09-01T06:18:32Z", "author_association": "MEMBER", "body": "Good suggestion, I'll setup a demo somewhere.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 629473827, "label": "Set up a demo"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/pocket-to-sqlite/issues/3#issuecomment-684424396", "issue_url": "https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/3", "id": 684424396, "node_id": "MDEyOklzc3VlQ29tbWVudDY4NDQyNDM5Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-01T06:17:45Z", "updated_at": "2020-09-01T06:17:45Z", "author_association": "MEMBER", "body": "It looks like I could ignore the `image` column and synthesize a unique key from the data in the `images` column using `$item_id/$image_id`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 503243784, "label": "Extract images into separate tables"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/dogsheep-beta/issues/4#issuecomment-684395444", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/4", "id": 684395444, "node_id": "MDEyOklzc3VlQ29tbWVudDY4NDM5NTQ0NA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-01T06:00:03Z", "updated_at": "2020-09-01T06:00:03Z", "author_association": "MEMBER", "body": "I ran `sqlite-utils optimize beta.db` against my test DB and the size reduced from 183M to 176M - and a 450ms search ran in 359ms. So not a huge improvement but still worthwhile.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 689839399, "label": "Optimize the FTS table"}, "performed_via_github_app": null} {"html_url": "https://github.com/dogsheep/dogsheep-beta/issues/3#issuecomment-684250044", "issue_url": "https://api.github.com/repos/dogsheep/dogsheep-beta/issues/3", "id": 684250044, "node_id": "MDEyOklzc3VlQ29tbWVudDY4NDI1MDA0NA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-09-01T05:01:09Z", "updated_at": "2020-09-01T05:01:23Z", "author_association": "MEMBER", "body": "Maybe this starts out as a custom templated canned query.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 689810340, "label": "Datasette plugin to provide custom page for running faceted, ranked searches"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/147#issuecomment-683528149", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/147", "id": 683528149, "node_id": "MDEyOklzc3VlQ29tbWVudDY4MzUyODE0OQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-08-31T03:17:26Z", "updated_at": "2020-08-31T03:17:26Z", "author_association": "OWNER", "body": "+1 to making this something that users can customize. An optional argument to the `Database` constructor would be a neat way to do this.\r\n\r\nI think there's a terrifying way that we could find this value... we could perform a binary search for it! Open up a memory connection and try running different bulk inserts against it and catch the exceptions - then adjust and try again.\r\n\r\nMy hunch is that we could perform just 2 or 3 probes (maybe against carefully selected values) to find the highest value that works. If this process took less than a few ms to run I'd be happy to do it automatically when the class is instantiated (and let users disable that automatic proving by setting a value using the constructor argument).", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 688670158, "label": "SQLITE_MAX_VARS maybe hard-coded too low"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/948#issuecomment-683448569", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/948", "id": 683448569, "node_id": "MDEyOklzc3VlQ29tbWVudDY4MzQ0ODU2OQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-08-30T17:39:09Z", "updated_at": "2020-08-30T18:34:34Z", "author_association": "OWNER", "body": "So the steps needed are:\r\n\r\n- Download and extract latest CodeMirror zip file\r\n- Rename `lib/codemirror.js` to `codemirror-5.57.0.js`\r\n- Rename `lib/codemirror.css` to `codemirror-5.57.0.css`\r\n- Rename `mode/sql/sql.js` to `codemirror-5.57.0-sql.js`\r\n- Edit both JS files to make the top comment a `/* */` block\r\n- Minify JavaScript files like this:\r\n - `npx uglify-js codemirror-5.57.0.js -o codemirror-5.57.0.min.js --comments '/LICENSE/'`\r\n - `npx uglify-js codemirror-5.57.0-sql.js -o codemirror-5.57.0-sql.min.js --comments '/LICENSE/'`\r\n- Check that the LICENSE comment did indeed survive minification\r\n- Minify CSS file like this:\r\n - `npx clean-css-cli codemirror-5.57.0.css -o codemirror-5.57.0.min.css`\r\n- Edit the `_codemirror.html` template to reference the new files\r\n- `git rm` the old files, `git add` the new files", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 684925907, "label": "Upgrade CodeMirror"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/948#issuecomment-683452613", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/948", "id": 683452613, "node_id": "MDEyOklzc3VlQ29tbWVudDY4MzQ1MjYxMw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-08-30T18:16:28Z", "updated_at": "2020-08-30T18:16:28Z", "author_association": "OWNER", "body": "I added documentation on how to upgrade CodeMirror for the future here: https://docs.datasette.io/en/latest/contributing.html#upgrading-codemirror", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 684925907, "label": "Upgrade CodeMirror"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/655#issuecomment-683449837", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/655", "id": 683449837, "node_id": "MDEyOklzc3VlQ29tbWVudDY4MzQ0OTgzNw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-08-30T17:51:38Z", "updated_at": "2020-08-30T17:51:38Z", "author_association": "OWNER", "body": "I think was fixed by #948", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 542553350, "label": "Copy and paste doesn't work reliably on iPhone for SQL editor"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/948#issuecomment-683449804", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/948", "id": 683449804, "node_id": "MDEyOklzc3VlQ29tbWVudDY4MzQ0OTgwNA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-08-30T17:51:18Z", "updated_at": "2020-08-30T17:51:18Z", "author_association": "OWNER", "body": "Copy and paste on mobile safari seems to work now. #655 ", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 684925907, "label": "Upgrade CodeMirror"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/948#issuecomment-683448635", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/948", "id": 683448635, "node_id": "MDEyOklzc3VlQ29tbWVudDY4MzQ0ODYzNQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-08-30T17:39:54Z", "updated_at": "2020-08-30T17:39:54Z", "author_association": "OWNER", "body": "I'll wait for this to deploy to https://latest.datasette.io/ and then test it in various desktop and mobile browsers.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 684925907, "label": "Upgrade CodeMirror"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/948#issuecomment-683445704", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/948", "id": 683445704, "node_id": "MDEyOklzc3VlQ29tbWVudDY4MzQ0NTcwNA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-08-30T17:11:58Z", "updated_at": "2020-08-30T17:33:30Z", "author_association": "OWNER", "body": "One catch: this stripped the license information from the top of the JS.\r\n\r\nI fixed this by editing the license to be a single `/* ... */` block comment instead of multiple `//` lines and running this:\r\n\r\n npx uglify-js codemirror-5.57.0.js -o codemirror-5.57.0.min.js --comments '/LICENSE/'\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 684925907, "label": "Upgrade CodeMirror"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/948#issuecomment-683445114", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/948", "id": 683445114, "node_id": "MDEyOklzc3VlQ29tbWVudDY4MzQ0NTExNA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-08-30T17:06:39Z", "updated_at": "2020-08-30T17:06:39Z", "author_association": "OWNER", "body": "Minifying using `npx`:\r\n```\r\nnpx uglify-js codemirror-5.57.0.js -o codemirror-5.57.0.min.js\r\nnpx uglify-js codemirror-5.57.0-sql.js -o codemirror-5.57.0-sql.min.js\r\nnpx clean-css-cli codemirror-5.57.0.css -o codemirror-5.57.0.min.css\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 684925907, "label": "Upgrade CodeMirror"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/145#issuecomment-683382252", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/145", "id": 683382252, "node_id": "MDEyOklzc3VlQ29tbWVudDY4MzM4MjI1Mg==", "user": {"value": 96218, "label": "simonwiles"}, "created_at": "2020-08-30T06:27:25Z", "updated_at": "2020-08-30T06:27:52Z", "author_association": "CONTRIBUTOR", "body": "Note: had to adjust the test above because trying to exhaust a `SQLITE_MAX_VARIABLE_NUMBER` of 250000 in 99 records requires 2526 columns, and trips the ` \"Rows can have a maximum of {} columns\".format(SQLITE_MAX_VARS)` check even before it trips the default `SQLITE_MAX_COLUMN` value (2000).", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 688659182, "label": "Bug when first record contains fewer columns than subsequent records"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/957#issuecomment-683357092", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/957", "id": 683357092, "node_id": "MDEyOklzc3VlQ29tbWVudDY4MzM1NzA5Mg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-08-30T00:15:51Z", "updated_at": "2020-08-30T00:16:02Z", "author_association": "OWNER", "body": "Weirdly even removing this single `datasette` import from `utils/asgi.py` didn't fix the circular import:\r\n\r\nhttps://github.com/simonw/datasette/blob/44cf424a94a85b74552075272660bb96a7432661/datasette/utils/asgi.py#L1-L3", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 688622148, "label": "Simplify imports of common classes"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/957#issuecomment-683356440", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/957", "id": 683356440, "node_id": "MDEyOklzc3VlQ29tbWVudDY4MzM1NjQ0MA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-08-30T00:08:18Z", "updated_at": "2020-08-30T00:10:26Z", "author_association": "OWNER", "body": "Annoyingly this seems to be the line that causes the circular import:\r\n```python\r\nfrom .utils.asgi import Forbidden, NotFound, Response\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 688622148, "label": "Simplify imports of common classes"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/957#issuecomment-683355993", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/957", "id": 683355993, "node_id": "MDEyOklzc3VlQ29tbWVudDY4MzM1NTk5Mw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-08-30T00:02:11Z", "updated_at": "2020-08-30T00:04:18Z", "author_association": "OWNER", "body": "I tried doing this and got this error:\r\n```\r\n(datasette) datasette % pytest\r\n==================================================================== test session starts =====================================================================\r\nplatform darwin -- Python 3.8.5, pytest-6.0.1, py-1.9.0, pluggy-0.13.1\r\nrootdir: /Users/simon/Dropbox/Development/datasette, configfile: pytest.ini\r\nplugins: asyncio-0.14.0, timeout-1.4.2\r\ncollected 1 item / 23 errors \r\n\r\n=========================================================================== ERRORS ===========================================================================\r\n_____________________________________________________________ ERROR collecting tests/test_api.py _____________________________________________________________\r\nImportError while importing test module '/Users/simon/Dropbox/Development/datasette/tests/test_api.py'.\r\nHint: make sure your test modules/packages have valid Python names.\r\nTraceback:\r\n/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/importlib/__init__.py:127: in import_module\r\n return _bootstrap._gcd_import(name[level:], package, level)\r\ntests/test_api.py:1: in \r\n from datasette.plugins import DEFAULT_PLUGINS\r\ndatasette/__init__.py:2: in \r\n from .app import Datasette\r\ndatasette/app.py:30: in \r\n from .views.base import DatasetteError, ureg\r\ndatasette/views/base.py:12: in \r\n from datasette.plugins import pm\r\ndatasette/plugins.py:26: in \r\n mod = importlib.import_module(plugin)\r\n/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/importlib/__init__.py:127: in import_module\r\n return _bootstrap._gcd_import(name[level:], package, level)\r\ndatasette/publish/heroku.py:2: in \r\n from datasette import hookimpl\r\nE ImportError: cannot import name 'hookimpl' from partially initialized module 'datasette' (most likely due to a circular import) (/Users/simon/Dropbox/Development/datasette/datasette/__init__.py)\r\n```\r\nThat's with `datasette/__init__.py` looking like this:\r\n```python\r\nfrom datasette.version import __version_info__, __version__ # noqa\r\nfrom .app import Datasette\r\nfrom .utils.asgi import Forbidden, NotFound, Response\r\nfrom .utils import actor_matches_allow, QueryInterrupted\r\nfrom .hookspecs import hookimpl # noqa\r\nfrom .hookspecs import hookspec # noqa\r\n\r\n\r\n__all__ = [\r\n \"actor_matches_allow\",\r\n \"hookimpl\",\r\n \"hookspec\",\r\n \"QueryInterrupted\",\r\n \"Forbidden\",\r\n \"NotFound\",\r\n \"Response\",\r\n \"Datasette\",\r\n]\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 688622148, "label": "Simplify imports of common classes"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/957#issuecomment-683355598", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/957", "id": 683355598, "node_id": "MDEyOklzc3VlQ29tbWVudDY4MzM1NTU5OA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-08-29T23:55:10Z", "updated_at": "2020-08-29T23:55:34Z", "author_association": "OWNER", "body": "Of these I think I'm going to promote the following to being importable directly `from datasette`:\r\n\r\n- `from datasette.app import Datasette`\r\n- `from datasette.utils import QueryInterrupted`\r\n- `from datasette.utils.asgi import Response, Forbidden, NotFound`\r\n- `from datasette.utils import actor_matches_allow`\r\n\r\nAll of the rest are infrequently used enough (or clearly named enough) that I'm happy to leave them as-is.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 688622148, "label": "Simplify imports of common classes"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/957#issuecomment-683355508", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/957", "id": 683355508, "node_id": "MDEyOklzc3VlQ29tbWVudDY4MzM1NTUwOA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-08-29T23:54:01Z", "updated_at": "2020-08-29T23:54:01Z", "author_association": "OWNER", "body": "Reviewing https://github.com/search?q=user%3Asimonw+%22from+datasette%22&type=Code I spotted these others:\r\n\r\n```python\r\n# Various:\r\nfrom datasette.utils import path_with_replaced_args\r\nfrom datasette.plugins import pm\r\nfrom datasette.utils import QueryInterrupted\r\nfrom datasette.utils.asgi import Response, Forbidden, NotFound\r\n\r\n# datasette-publish-vercel:\r\nfrom datasette.publish.common import (\r\n add_common_publish_arguments_and_options,\r\n fail_if_publish_binary_not_installed\r\n)\r\nfrom datasette.utils import temporary_docker_directory\r\n\r\n# datasette-insert\r\nfrom datasette.utils import actor_matches_allow, sqlite3\r\n\r\n# obsolete: russian-ira-facebook-ads-datasette \r\nfrom datasette.utils import TableFilter\r\n\r\n# simonw/museums\r\nfrom datasette.utils.asgi import asgi_send\r\n\r\n# datasette-media\r\nfrom datasette.utils.asgi import Response, asgi_send_file\r\n\r\n# datasette/tests/plugins/my_plugin.py\r\nfrom datasette.facets import Facet\r\n\r\n# datasette-graphql\r\nfrom datasette.views.table import TableView\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 688622148, "label": "Simplify imports of common classes"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/956#issuecomment-683214102", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/956", "id": 683214102, "node_id": "MDEyOklzc3VlQ29tbWVudDY4MzIxNDEwMg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-08-29T01:32:21Z", "updated_at": "2020-08-29T01:32:21Z", "author_association": "OWNER", "body": "Maybe the bug here is the double colon?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 688427751, "label": "Push to Docker Hub failed - but it shouldn't run for alpha releases anyway"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/956#issuecomment-683213973", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/956", "id": 683213973, "node_id": "MDEyOklzc3VlQ29tbWVudDY4MzIxMzk3Mw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-08-29T01:31:39Z", "updated_at": "2020-08-29T01:31:39Z", "author_association": "OWNER", "body": "Here's how the old Travis mechanism worked: https://github.com/simonw/datasette/blob/52eabb019d4051084b21524bd0fd9c2731126985/.travis.yml#L41-L47\r\n\r\nSo I was assuming that the eqivalent of `$REPO:$TRAVIS_TAG` in GitHub Actions is `$REPO::${GITHUB_REF#refs/tags/}`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 688427751, "label": "Push to Docker Hub failed - but it shouldn't run for alpha releases anyway"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/956#issuecomment-683212960", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/956", "id": 683212960, "node_id": "MDEyOklzc3VlQ29tbWVudDY4MzIxMjk2MA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-08-29T01:25:34Z", "updated_at": "2020-08-29T01:25:34Z", "author_association": "OWNER", "body": "So I guess this bit is wrong: https://github.com/simonw/datasette/blob/c36e287d71d68ecb2a45e9808eede15f19f931fb/.github/workflows/publish.yml#L71-L73", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 688427751, "label": "Push to Docker Hub failed - but it shouldn't run for alpha releases anyway"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/956#issuecomment-683212421", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/956", "id": 683212421, "node_id": "MDEyOklzc3VlQ29tbWVudDY4MzIxMjQyMQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-08-29T01:22:23Z", "updated_at": "2020-08-29T01:22:23Z", "author_association": "OWNER", "body": "Here's the error message again:\r\n\r\n> invalid argument `\"***/datasette::0.49a0\"` for `\"-t, --tag\"` flag: invalid reference format", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 688427751, "label": "Push to Docker Hub failed - but it shouldn't run for alpha releases anyway"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/956#issuecomment-683212246", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/956", "id": 683212246, "node_id": "MDEyOklzc3VlQ29tbWVudDY4MzIxMjI0Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-08-29T01:21:26Z", "updated_at": "2020-08-29T01:21:26Z", "author_association": "OWNER", "body": "I added this but I have no idea if I got it right or not: https://github.com/simonw/datasette/blob/c36e287d71d68ecb2a45e9808eede15f19f931fb/.github/workflows/publish.yml#L58-L63", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 688427751, "label": "Push to Docker Hub failed - but it shouldn't run for alpha releases anyway"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/955#issuecomment-683189334", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/955", "id": 683189334, "node_id": "MDEyOklzc3VlQ29tbWVudDY4MzE4OTMzNA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-08-28T23:30:48Z", "updated_at": "2020-08-28T23:30:48Z", "author_association": "OWNER", "body": "Also https://github.com/simonw/datasette-copyable", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 687711713, "label": "Release updated datasette-atom and datasette-ics"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/955#issuecomment-683185861", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/955", "id": 683185861, "node_id": "MDEyOklzc3VlQ29tbWVudDY4MzE4NTg2MQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-08-28T23:17:09Z", "updated_at": "2020-08-28T23:17:09Z", "author_association": "OWNER", "body": "I released 0.49a0 which means I can update the main branches of those two plugins - I'll push a release of them once 0.49 is fully shipped.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 687711713, "label": "Release updated datasette-atom and datasette-ics"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/144#issuecomment-683180581", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/144", "id": 683180581, "node_id": "MDEyOklzc3VlQ29tbWVudDY4MzE4MDU4MQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-08-28T22:57:04Z", "updated_at": "2020-08-28T22:57:04Z", "author_association": "OWNER", "body": "That worked! https://github.com/simonw/sqlite-utils/runs/1043640785?check_suite_focus=true\r\n\r\n\"Add_numpy_to_the_matrix__refs__144_\u00b7_simonw_sqlite-utils_cbc22ef\"\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 688395275, "label": "Run some tests against numpy"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/144#issuecomment-683179678", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/144", "id": 683179678, "node_id": "MDEyOklzc3VlQ29tbWVudDY4MzE3OTY3OA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-08-28T22:53:17Z", "updated_at": "2020-08-28T22:53:17Z", "author_association": "OWNER", "body": "I'm going to try doing this as a GitHub Actions test matrix.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 688395275, "label": "Run some tests against numpy"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/139#issuecomment-683178570", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/139", "id": 683178570, "node_id": "MDEyOklzc3VlQ29tbWVudDY4MzE3ODU3MA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-08-28T22:48:51Z", "updated_at": "2020-08-28T22:48:51Z", "author_association": "OWNER", "body": "Thanks @simonwiles, this is now released in 2.16.1: https://sqlite-utils.readthedocs.io/en/stable/changelog.html", "reactions": "{\"total_count\": 2, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 1, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 686978131, "label": "insert_all(..., alter=True) should work for new columns introduced after the first 100 records"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/143#issuecomment-683175491", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/143", "id": 683175491, "node_id": "MDEyOklzc3VlQ29tbWVudDY4MzE3NTQ5MQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-08-28T22:37:15Z", "updated_at": "2020-08-28T22:37:15Z", "author_association": "OWNER", "body": "I'm going to start running black exclusively in the GitHub Actions workflow, rather than having it run by the unit tests themselves.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 688389933, "label": "Move to GitHub Actions CI"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/142#issuecomment-683173375", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/142", "id": 683173375, "node_id": "MDEyOklzc3VlQ29tbWVudDY4MzE3MzM3NQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-08-28T22:29:02Z", "updated_at": "2020-08-28T22:29:02Z", "author_association": "OWNER", "body": "Yeah I think that failure is actually because there's a brand new release of Black out and it subtly changes some of the formatting rules. I'll merge this and then run Black against the entire codebase.", "reactions": "{\"total_count\": 1, \"+1\": 1, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 688386219, "label": "insert_all(..., alter=True) should work for new columns introduced after the first 100 records"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/142#issuecomment-683172829", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/142", "id": 683172829, "node_id": "MDEyOklzc3VlQ29tbWVudDY4MzE3MjgyOQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-08-28T22:27:05Z", "updated_at": "2020-08-28T22:27:05Z", "author_association": "OWNER", "body": "Looks like it failed the \"black\" formatting test - possibly because there's a new release if black out. I'm going to merge despite that failure.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 688386219, "label": "insert_all(..., alter=True) should work for new columns introduced after the first 100 records"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/pull/142#issuecomment-683172082", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/142", "id": 683172082, "node_id": "MDEyOklzc3VlQ29tbWVudDY4MzE3MjA4Mg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-08-28T22:24:25Z", "updated_at": "2020-08-28T22:24:25Z", "author_association": "OWNER", "body": "Thanks very much!", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 688386219, "label": "insert_all(..., alter=True) should work for new columns introduced after the first 100 records"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/119#issuecomment-683146200", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/119", "id": 683146200, "node_id": "MDEyOklzc3VlQ29tbWVudDY4MzE0NjIwMA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-08-28T21:05:37Z", "updated_at": "2020-08-28T21:05:37Z", "author_association": "OWNER", "body": "Maybe use `transform_table()` in #114 for this? Would be less efficient as it would copy the whole table but it would reduce library complexity a bit.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 652700770, "label": "Ability to remove a foreign key"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/139#issuecomment-682815377", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/139", "id": 682815377, "node_id": "MDEyOklzc3VlQ29tbWVudDY4MjgxNTM3Nw==", "user": {"value": 96218, "label": "simonwiles"}, "created_at": "2020-08-28T16:14:58Z", "updated_at": "2020-08-28T16:14:58Z", "author_association": "CONTRIBUTOR", "body": "Thanks! And yeah, I had updating the docs on my list too :) Will try to get to it this afternoon (budgeting time is fraught with uncertainty at the moment!).", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 686978131, "label": "insert_all(..., alter=True) should work for new columns introduced after the first 100 records"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/139#issuecomment-682771226", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/139", "id": 682771226, "node_id": "MDEyOklzc3VlQ29tbWVudDY4Mjc3MTIyNg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-08-28T15:57:42Z", "updated_at": "2020-08-28T15:57:42Z", "author_association": "OWNER", "body": "That pull request should update this section of the documentation too:\r\n\r\n> If you have more than one record to insert, the insert_all() method is a much more efficient way of inserting them. Just like insert() it will automatically detect the columns that should be created, but it will inspect the first batch of 100 items to help decide what those column types should be.\r\n\r\nhttps://github.com/simonw/sqlite-utils/blob/ea87c2b943fdd162c42a900ac0aea5ecc2f4b9d9/docs/python-api.rst#L393", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 686978131, "label": "insert_all(..., alter=True) should work for new columns introduced after the first 100 records"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/139#issuecomment-682762911", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/139", "id": 682762911, "node_id": "MDEyOklzc3VlQ29tbWVudDY4Mjc2MjkxMQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-08-28T15:54:57Z", "updated_at": "2020-08-28T15:55:20Z", "author_association": "OWNER", "body": "Here's a suggested test update:\r\n```diff\r\ndiff --git a/sqlite_utils/db.py b/sqlite_utils/db.py\r\nindex a8791c3..12fa2f2 100644\r\n--- a/sqlite_utils/db.py\r\n+++ b/sqlite_utils/db.py\r\n@@ -1074,6 +1074,13 @@ class Table(Queryable):\r\n all_columns = list(sorted(all_columns))\r\n if hash_id:\r\n all_columns.insert(0, hash_id)\r\n+ else:\r\n+ all_columns += [\r\n+ column\r\n+ for record in chunk\r\n+ for column in record\r\n+ if column not in all_columns\r\n+ ]\r\n validate_column_names(all_columns)\r\n first = False\r\n # values is the list of insert data that is passed to the\r\ndiff --git a/tests/test_create.py b/tests/test_create.py\r\nindex a84eb8d..3a7fafc 100644\r\n--- a/tests/test_create.py\r\n+++ b/tests/test_create.py\r\n@@ -707,13 +707,15 @@ def test_insert_thousands_using_generator(fresh_db):\r\n assert 10000 == fresh_db[\"test\"].count\r\n \r\n \r\n-def test_insert_thousands_ignores_extra_columns_after_first_100(fresh_db):\r\n+def test_insert_thousands_adds_extra_columns_after_first_100(fresh_db):\r\n+ # https://github.com/simonw/sqlite-utils/issues/139\r\n fresh_db[\"test\"].insert_all(\r\n [{\"i\": i, \"word\": \"word_{}\".format(i)} for i in range(100)]\r\n- + [{\"i\": 101, \"extra\": \"This extra column should cause an exception\"}]\r\n+ + [{\"i\": 101, \"extra\": \"Should trigger ALTER\"}],\r\n+ alter=True,\r\n )\r\n rows = fresh_db.execute_returning_dicts(\"select * from test where i = 101\")\r\n- assert [{\"i\": 101, \"word\": None}] == rows\r\n+ assert [{\"i\": 101, \"word\": None, \"extra\": \"Should trigger ALTER\"}] == rows\r\n \r\n \r\n def test_insert_ignore(fresh_db):\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 686978131, "label": "insert_all(..., alter=True) should work for new columns introduced after the first 100 records"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/954#issuecomment-682312736", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/954", "id": 682312736, "node_id": "MDEyOklzc3VlQ29tbWVudDY4MjMxMjczNg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-08-28T04:05:01Z", "updated_at": "2020-08-28T04:05:10Z", "author_association": "OWNER", "body": "> It can also return a dictionary with the following keys. This format is **deprecated** as-of Datasette 0.49 and will be removed by Datasette 1.0.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 687694947, "label": "Remove old register_output_renderer dict mechanism in Datasette 1.0"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/953#issuecomment-682312494", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/953", "id": 682312494, "node_id": "MDEyOklzc3VlQ29tbWVudDY4MjMxMjQ5NA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-08-28T04:03:56Z", "updated_at": "2020-08-28T04:03:56Z", "author_association": "OWNER", "body": "Documentation says that the old dictionary mechanism will be deprecated by 1.0:\r\n\r\nhttps://github.com/simonw/datasette/blob/799ecae94824640bdff21f86997f69844048d5c3/docs/plugin_hooks.rst#L460", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 687681018, "label": "register_output_renderer render function should be able to return a Response"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/139#issuecomment-682285212", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/139", "id": 682285212, "node_id": "MDEyOklzc3VlQ29tbWVudDY4MjI4NTIxMg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-08-28T02:12:51Z", "updated_at": "2020-08-28T02:12:51Z", "author_association": "OWNER", "body": "I'd be happy to accept a PR for this, provided it included updated unit tests that illustrate it working. I think this is a really good improvement.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 686978131, "label": "insert_all(..., alter=True) should work for new columns introduced after the first 100 records"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/139#issuecomment-682284908", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/139", "id": 682284908, "node_id": "MDEyOklzc3VlQ29tbWVudDY4MjI4NDkwOA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-08-28T02:11:40Z", "updated_at": "2020-08-28T02:11:40Z", "author_association": "OWNER", "body": "This is deliberate behaviour, but I'm not at all attached to it - you're right in pointing out that it's actually pretty unexpected.\r\n\r\nI'd be happy to change this behaviour so if you pass `alter=True` and then use `.insert_all()` on more than 100 rows it works as you would expect, instead of silently ignoring new columns past the first 100 rows. I don't expect that anyone would be depending on the current behaviour (ignore new columns after the first 100) such that this should be considered a backwards incompatible change.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 686978131, "label": "insert_all(..., alter=True) should work for new columns introduced after the first 100 records"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/139#issuecomment-682182178", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/139", "id": 682182178, "node_id": "MDEyOklzc3VlQ29tbWVudDY4MjE4MjE3OA==", "user": {"value": 96218, "label": "simonwiles"}, "created_at": "2020-08-27T20:46:18Z", "updated_at": "2020-08-27T20:46:18Z", "author_association": "CONTRIBUTOR", "body": "> I tried changing the batch_size argument to the total number of records, but it seems only to effect the number of rows that are committed at a time, and has no influence on this problem.\r\n\r\nSo the reason for this is that the `batch_size` for import is limited (of necessity) here: https://github.com/simonw/sqlite-utils/blob/main/sqlite_utils/db.py#L1048\r\n\r\nWith regard to the issue of ignoring columns, however, I made a fork and hacked a temporary fix that looks like this:\r\nhttps://github.com/simonwiles/sqlite-utils/commit/3901f43c6a712a1a3efc340b5b8d8fd0cbe8ee63\r\n\r\nIt doesn't seem to affect performance enormously (but I've not tested it thoroughly), and it now does what I need (and would expect, tbh), but it now fails the test here:\r\nhttps://github.com/simonw/sqlite-utils/blob/main/tests/test_create.py#L710-L716\r\n\r\nThe existence of this test suggests that `insert_all()` is behaving as intended, of course. It seems odd to me that this would be a desirable default behaviour (let alone the only behaviour), and its not very prominently flagged-up, either.\r\n\r\n@simonw is this something you'd be willing to look at a PR for? I assume you wouldn't want to change the default behaviour at this point, but perhaps an option could be provided, or at least a bit more of a warning in the docs. Are there oversights in the implementation that I've made?\r\n\r\nWould be grateful for your thoughts! Thanks!\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 686978131, "label": "insert_all(..., alter=True) should work for new columns introduced after the first 100 records"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/950#issuecomment-680374196", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/950", "id": 680374196, "node_id": "MDEyOklzc3VlQ29tbWVudDY4MDM3NDE5Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-08-26T00:43:50Z", "updated_at": "2020-08-26T00:43:50Z", "author_association": "OWNER", "body": "The problem with the term \"private\" is that it could be confused with the concept of databases that aren't visible to the public due to the permissions system - the ones that are displayed with the padlock icon e.g. on https://datasette-auth-passwords-demo.datasette.io/\r\n\r\n\"datasette-auth-passwords_demo__public__private\"\r\n\r\nSo I think \"secret\" is a better term for these.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 685806511, "label": "Private/secret databases: database files that are only visible to plugins"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/950#issuecomment-680264202", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/950", "id": 680264202, "node_id": "MDEyOklzc3VlQ29tbWVudDY4MDI2NDIwMg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-08-25T20:53:13Z", "updated_at": "2020-08-25T20:53:13Z", "author_association": "OWNER", "body": "Forcing people to spell out `datasette github.db --private private.db` isn't terrible though.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 685806511, "label": "Private/secret databases: database files that are only visible to plugins"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/950#issuecomment-680263999", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/950", "id": 680263999, "node_id": "MDEyOklzc3VlQ29tbWVudDY4MDI2Mzk5OQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-08-25T20:52:47Z", "updated_at": "2020-08-25T20:52:47Z", "author_association": "OWNER", "body": "Naming challenge: secret databases or private databases?\r\n\r\nI prefer private. But `datasette -p` is already taken by `--port`. `datasette -s` is currently available.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 685806511, "label": "Private/secret databases: database files that are only visible to plugins"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/950#issuecomment-680263427", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/950", "id": 680263427, "node_id": "MDEyOklzc3VlQ29tbWVudDY4MDI2MzQyNw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-08-25T20:51:30Z", "updated_at": "2020-08-25T20:52:13Z", "author_association": "OWNER", "body": "`datasette-graphql` currently dispatches requests through the `TableView` class, so if that couldn't access private databases then it would not be able to either. See also the concept for `datasette.get(...)` as an internal API in #943 - that might need to have a mechanism for also being able to query private databases, maybe `datasette.get(path, allow_private=True)`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 685806511, "label": "Private/secret databases: database files that are only visible to plugins"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/949#issuecomment-679367931", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/949", "id": 679367931, "node_id": "MDEyOklzc3VlQ29tbWVudDY3OTM2NzkzMQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-08-24T21:09:50Z", "updated_at": "2020-08-24T21:09:50Z", "author_association": "OWNER", "body": "I'm attracted to this because of how good GraphiQL is for auto-completing queries. But I realize there's a problem here: GraphQL is designed to be autocomplete-friendly, but SQL is not. If you type `select ` and it doesn't know what's going in the `from` clause it can't give you good column autocomplete, for example.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 684961449, "label": "Try out CodeMirror SQL hints"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/949#issuecomment-679363710", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/949", "id": 679363710, "node_id": "MDEyOklzc3VlQ29tbWVudDY3OTM2MzcxMA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-08-24T21:00:43Z", "updated_at": "2020-08-24T21:00:43Z", "author_association": "OWNER", "body": "I think this requires three extra files from https://github.com/codemirror/CodeMirror/tree/5.57.0/addon/hint\r\n\r\n- `show-hint.css`\r\n- `show-hint.js`\r\n- `sql-hint.js`\r\n\r\n", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 684961449, "label": "Try out CodeMirror SQL hints"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/948#issuecomment-679355426", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/948", "id": 679355426, "node_id": "MDEyOklzc3VlQ29tbWVudDY3OTM1NTQyNg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-08-24T20:43:07Z", "updated_at": "2020-08-24T20:43:07Z", "author_association": "OWNER", "body": "It would also be interesting to try out the SQL hint mode, which can autocomplete against tables and columns. This demo shows how to configure that: https://codemirror.net/mode/sql/", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 684925907, "label": "Upgrade CodeMirror"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/948#issuecomment-679333717", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/948", "id": 679333717, "node_id": "MDEyOklzc3VlQ29tbWVudDY3OTMzMzcxNw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-08-24T19:55:59Z", "updated_at": "2020-08-24T19:55:59Z", "author_association": "OWNER", "body": "CodeMirror 6 is in pre-release at the moment and is a complete rewrite. I'll stick with the 5.x series for now. https://github.com/codemirror/codemirror.next/", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 684925907, "label": "Upgrade CodeMirror"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/138#issuecomment-678732667", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/138", "id": 678732667, "node_id": "MDEyOklzc3VlQ29tbWVudDY3ODczMjY2Nw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-08-23T05:46:10Z", "updated_at": "2020-08-23T05:46:10Z", "author_association": "OWNER", "body": "Actually the `TEXT` column thing wasn't a `sqlite-utils` issue, it was unique to how `shapefile-to-spatialite` was creating the table when using the SpatiaLite extension.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 684118950, "label": "extracts= doesn't configure foreign keys"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/136#issuecomment-678508056", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/136", "id": 678508056, "node_id": "MDEyOklzc3VlQ29tbWVudDY3ODUwODA1Ng==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-08-21T21:13:41Z", "updated_at": "2020-08-21T21:13:41Z", "author_association": "OWNER", "body": "The `--spatialite` option should be available for other useful commands too, refs #137.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 683812642, "label": "--load-extension=spatialite shortcut option"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/137#issuecomment-678507502", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/137", "id": 678507502, "node_id": "MDEyOklzc3VlQ29tbWVudDY3ODUwNzUwMg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-08-21T21:13:19Z", "updated_at": "2020-08-21T21:13:19Z", "author_association": "OWNER", "body": "Adding `--spatialite` too would be great for usability: #136", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 683830416, "label": "--load-extension for other sqlite-utils commands"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/134#issuecomment-678497497", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/134", "id": 678497497, "node_id": "MDEyOklzc3VlQ29tbWVudDY3ODQ5NzQ5Nw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-08-21T21:06:26Z", "updated_at": "2020-08-21T21:06:26Z", "author_association": "OWNER", "body": "Ended up needing two skipIfs:\r\n\r\nhttps://github.com/simonw/sqlite-utils/blob/7e9aad7e1c09d1cf80d0b4d17d6157212a4b857d/tests/test_cli.py#L888-L893", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 683804172, "label": "--load-extension option for sqlite-utils query"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/136#issuecomment-678480969", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/136", "id": 678480969, "node_id": "MDEyOklzc3VlQ29tbWVudDY3ODQ4MDk2OQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-08-21T20:33:45Z", "updated_at": "2020-08-21T20:33:45Z", "author_association": "OWNER", "body": "I think this should initialize SpatiaLite against the current database if it has not been initialized already.\r\n\r\nRelevant code: https://github.com/simonw/shapefile-to-sqlite/blob/e754d0747ca2facf9a7433e2d5d15a6a37a9cf6e/shapefile_to_sqlite/utils.py#L112-L126\r\n\r\n```python\r\ndef init_spatialite(db, lib):\r\n db.conn.enable_load_extension(True)\r\n db.conn.load_extension(lib)\r\n # Initialize SpatiaLite if not yet initialized\r\n if \"spatial_ref_sys\" in db.table_names():\r\n return\r\n db.conn.execute(\"select InitSpatialMetadata(1)\")\r\n\r\n\r\ndef ensure_table_has_geometry(db, table, table_srid):\r\n if \"geometry\" not in db[table].columns_dict:\r\n db.conn.execute(\r\n \"SELECT AddGeometryColumn(?, 'geometry', ?, 'GEOMETRY', 2);\",\r\n [table, table_srid],\r\n )\r\n```\r\nNot sure if I should add a utility function or CLI command for that `ensure_table_has_geometry` bit.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 683812642, "label": "--load-extension=spatialite shortcut option"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/135#issuecomment-678479741", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/135", "id": 678479741, "node_id": "MDEyOklzc3VlQ29tbWVudDY3ODQ3OTc0MQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-08-21T20:30:37Z", "updated_at": "2020-08-21T20:30:37Z", "author_association": "OWNER", "body": "Docs: https://github.com/simonw/sqlite-utils/blob/bf4c6b7c82fab6b2400e48424f8dac1ae2f0a2dc/docs/python-api.rst#finding-spatialite", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 683805434, "label": "Code for finding SpatiaLite in the usual locations"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/135#issuecomment-678476842", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/135", "id": 678476842, "node_id": "MDEyOklzc3VlQ29tbWVudDY3ODQ3Njg0Mg==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-08-21T20:23:13Z", "updated_at": "2020-08-21T20:23:13Z", "author_association": "OWNER", "body": "I'm going to start with just the first two - I'm not convinced I understand the `.so.5` variants.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 683805434, "label": "Code for finding SpatiaLite in the usual locations"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/134#issuecomment-678476338", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/134", "id": 678476338, "node_id": "MDEyOklzc3VlQ29tbWVudDY3ODQ3NjMzOA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-08-21T20:22:02Z", "updated_at": "2020-08-21T20:22:02Z", "author_association": "OWNER", "body": "I think that adds it as `/usr/lib/x86_64-linux-gnu/mod_spatialite.so`.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 683804172, "label": "--load-extension option for sqlite-utils query"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/135#issuecomment-678475578", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/135", "id": 678475578, "node_id": "MDEyOklzc3VlQ29tbWVudDY3ODQ3NTU3OA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-08-21T20:20:05Z", "updated_at": "2020-08-21T20:20:05Z", "author_association": "OWNER", "body": "https://github.com/simonw/cryptozoology/blob/2ad69168f3b78ebd90a2cbeea8136c9115e2a9b7/build_cryptids_database.py#L16-L22\r\n```python\r\ntry_these = (\r\n \"mod_spatialite\",\r\n \"/usr/local/lib/mod_spatialite.dylib\",\r\n \"/usr/lib/x86_64-linux-gnu/mod_spatialite.so\",\r\n \"/usr/lib/x86_64-linux-gnu/libspatialite.so.5\",\r\n \"/usr/lib/x86_64-linux-gnu/libspatialite.so.7\",\r\n)\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 683805434, "label": "Code for finding SpatiaLite in the usual locations"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/134#issuecomment-678474928", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/134", "id": 678474928, "node_id": "MDEyOklzc3VlQ29tbWVudDY3ODQ3NDkyOA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-08-21T20:18:33Z", "updated_at": "2020-08-21T20:18:33Z", "author_association": "OWNER", "body": "This should get me SpatiaLite in the GitHub Actions Ubuntu:\r\n```\r\napt install libsqlite3-mod-spatialite\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 683804172, "label": "--load-extension option for sqlite-utils query"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/sqlite-utils/issues/134#issuecomment-678474018", "issue_url": "https://api.github.com/repos/simonw/sqlite-utils/issues/134", "id": 678474018, "node_id": "MDEyOklzc3VlQ29tbWVudDY3ODQ3NDAxOA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-08-21T20:16:20Z", "updated_at": "2020-08-21T20:16:20Z", "author_association": "OWNER", "body": "Trickiest part of this is how to write a test for it.\r\n\r\nI'll do a `pytest.skipIf` that only executes the test if SpatiaLite is available.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 683804172, "label": "--load-extension option for sqlite-utils query"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/945#issuecomment-676556377", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/945", "id": 676556377, "node_id": "MDEyOklzc3VlQ29tbWVudDY3NjU1NjM3Nw==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-08-19T17:21:16Z", "updated_at": "2020-08-19T17:21:16Z", "author_association": "OWNER", "body": "Documented here: https://docs.datasette.io/en/latest/plugins.html#installing-plugins", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 682005535, "label": "datasette install -U for upgrading packages"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/943#issuecomment-675889865", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/943", "id": 675889865, "node_id": "MDEyOklzc3VlQ29tbWVudDY3NTg4OTg2NQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-08-19T06:57:00Z", "updated_at": "2020-08-19T06:57:00Z", "author_association": "OWNER", "body": "Maybe `.get` vs `.get_html`?", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 681375466, "label": "await datasette.client.get(path) mechanism for executing internal requests"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/943#issuecomment-675889551", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/943", "id": 675889551, "node_id": "MDEyOklzc3VlQ29tbWVudDY3NTg4OTU1MQ==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-08-19T06:56:06Z", "updated_at": "2020-08-19T06:56:17Z", "author_association": "OWNER", "body": "I'm leaning towards defaulting to JSON as the requested format - you can pass `format=\"html\"` if you want HTML.\r\n\r\nBut weird that it's different from the web UI.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 681375466, "label": "await datasette.client.get(path) mechanism for executing internal requests"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/943#issuecomment-675884980", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/943", "id": 675884980, "node_id": "MDEyOklzc3VlQ29tbWVudDY3NTg4NDk4MA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-08-19T06:44:26Z", "updated_at": "2020-08-19T06:44:26Z", "author_association": "OWNER", "body": "Need to decide what to do about JSON responses.\r\n\r\nWhen called from a template it's likely the intent will be to further loop through the JSON data returned. It would be annoying to have to run `json.loads` here.\r\n\r\nMaybe a `.get_json()` method then? Or even return a response that has `.json()` and `.text` similar to `httpx` - or just return an `httpx` response.", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 681375466, "label": "await datasette.client.get(path) mechanism for executing internal requests"}, "performed_via_github_app": null} {"html_url": "https://github.com/simonw/datasette/issues/944#issuecomment-675830678", "issue_url": "https://api.github.com/repos/simonw/datasette/issues/944", "id": 675830678, "node_id": "MDEyOklzc3VlQ29tbWVudDY3NTgzMDY3OA==", "user": {"value": 9599, "label": "simonw"}, "created_at": "2020-08-19T03:30:10Z", "updated_at": "2020-08-19T03:30:10Z", "author_association": "OWNER", "body": "These templates will need a way to raise a 404 - so that if the template itself is deciding if the page exists (for example using `datasette-template-sql` or the proposed `datasette.get()` method from #943 or the `graphql()` template function in https://github.com/simonw/datasette-graphql/issues/50) it can return a regular 404 page.\r\n\r\nThis can imitate the `custom_redirect()` function from https://docs.datasette.io/en/stable/custom_templates.html#custom-redirects:\r\n```html+jinja\r\n{{ custom_redirect(\"https://github.com/simonw/datasette\", 301) }}\r\n```\r\nIt could be as simple as this:\r\n```\r\n{{ raise_404(\"Museum not found\") }}\r\n```", "reactions": "{\"total_count\": 0, \"+1\": 0, \"-1\": 0, \"laugh\": 0, \"hooray\": 0, \"confused\": 0, \"heart\": 0, \"rocket\": 0, \"eyes\": 0}", "issue": {"value": 681516976, "label": "Path parameters for custom pages"}, "performed_via_github_app": null}