html_url,issue_url,id,node_id,user,created_at,updated_at,author_association,body,reactions,issue,performed_via_github_app https://github.com/simonw/sqlite-utils/issues/149#issuecomment-688482355,https://api.github.com/repos/simonw/sqlite-utils/issues/149,688482355,MDEyOklzc3VlQ29tbWVudDY4ODQ4MjM1NQ==,9599,2020-09-07T19:22:51Z,2020-09-07T19:22:51Z,OWNER,"And the SQLite documentation says: > When the REPLACE conflict resolution strategy deletes rows in order to satisfy a constraint, [delete triggers](https://www.sqlite.org/lang_createtrigger.html) fire if and only if [recursive triggers](https://www.sqlite.org/pragma.html#pragma_recursive_triggers) are enabled.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",695319258, https://github.com/simonw/sqlite-utils/issues/149#issuecomment-688482055,https://api.github.com/repos/simonw/sqlite-utils/issues/149,688482055,MDEyOklzc3VlQ29tbWVudDY4ODQ4MjA1NQ==,9599,2020-09-07T19:21:42Z,2020-09-07T19:21:42Z,OWNER,"Using `replace=True` there executes `INSERT OR REPLACE` - and Dan Kennedy (SQLite maintainer) on the SQLite forums said this: > Are you using ""REPLACE INTO"", or ""UPDATE OR REPLACE"" on the ""licenses"" table without having first executed ""PRAGMA recursive_triggers = 1""? The docs note that delete triggers will not be fired in this case, which would explain things. Second paragraph under ""REPLACE"" here: > > https://www.sqlite.org/lang_conflict.html","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",695319258, https://github.com/simonw/sqlite-utils/issues/149#issuecomment-688481374,https://api.github.com/repos/simonw/sqlite-utils/issues/149,688481374,MDEyOklzc3VlQ29tbWVudDY4ODQ4MTM3NA==,9599,2020-09-07T19:19:08Z,2020-09-07T19:19:08Z,OWNER,"reading through the code for `github-to-sqlite repos` - one of the things it does is calls `save_license` for each repo: https://github.com/dogsheep/github-to-sqlite/blob/39b2234253096bd579feed4e25104698b8ccd2ba/github_to_sqlite/utils.py#L259-L262 ```python def save_license(db, license): if license is None: return None return db[""licenses""].insert(license, pk=""key"", replace=True).last_pk ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",695319258, https://github.com/simonw/sqlite-utils/issues/149#issuecomment-688480665,https://api.github.com/repos/simonw/sqlite-utils/issues/149,688480665,MDEyOklzc3VlQ29tbWVudDY4ODQ4MDY2NQ==,9599,2020-09-07T19:16:20Z,2020-09-07T19:16:20Z,OWNER,"Aha! I have managed to replicate the bug: ``` (github-to-sqlite) /tmp % sqlite-utils tables --counts github.db | grep licenses {""table"": ""licenses"", ""count"": 7}, {""table"": ""licenses_fts_data"", ""count"": 35}, {""table"": ""licenses_fts_idx"", ""count"": 16}, {""table"": ""licenses_fts_docsize"", ""count"": 9151}, {""table"": ""licenses_fts_config"", ""count"": 1}, {""table"": ""licenses_fts"", ""count"": 7}, (github-to-sqlite) /tmp % github-to-sqlite repos github.db dogsheep (github-to-sqlite) /tmp % sqlite-utils tables --counts github.db | grep licenses {""table"": ""licenses"", ""count"": 7}, {""table"": ""licenses_fts_data"", ""count"": 45}, {""table"": ""licenses_fts_idx"", ""count"": 26}, {""table"": ""licenses_fts_docsize"", ""count"": 9161}, {""table"": ""licenses_fts_config"", ""count"": 1}, {""table"": ""licenses_fts"", ""count"": 7}, ``` Note that the number of records in `licenses_fts_docsize` went from 9151 to 9161.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",695319258, https://github.com/simonw/sqlite-utils/issues/149#issuecomment-688464181,https://api.github.com/repos/simonw/sqlite-utils/issues/149,688464181,MDEyOklzc3VlQ29tbWVudDY4ODQ2NDE4MQ==,9599,2020-09-07T18:19:54Z,2020-09-07T18:19:54Z,OWNER,"Even though that table doesn't declare an integer primary key it does have a `rowid` column: https://github-to-sqlite.dogsheep.net/github?sql=select+rowid%2C+%5Bkey%5D%2C+name%2C+spdx_id%2C+url%2C+node_id+from+licenses+order+by+%5Bkey%5D+limit+101 | rowid | key | name | spdx_id | url | node_id | | --- | --- | --- | --- | --- | --- | | 9150 | apache-2.0 | Apache License 2.0 | Apache-2.0 | | MDc6TGljZW5zZTI= | | 112 | bsd-3-clause | BSD 3-Clause ""New"" or ""Revised"" License | BSD-3-Clause | | MDc6TGljZW5zZTU= | https://www.sqlite.org/rowidtable.html explains has this clue: > If the rowid is not aliased by INTEGER PRIMARY KEY then it is not persistent and might change. In particular the VACUUM command will change rowids for tables that do not declare an INTEGER PRIMARY KEY. Therefore, applications should not normally access the rowid directly, but instead use an INTEGER PRIMARY KEY. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",695319258, https://github.com/simonw/sqlite-utils/issues/149#issuecomment-688460865,https://api.github.com/repos/simonw/sqlite-utils/issues/149,688460865,MDEyOklzc3VlQ29tbWVudDY4ODQ2MDg2NQ==,9599,2020-09-07T18:07:14Z,2020-09-07T18:07:14Z,OWNER,"Another likely culprit: `licenses` has a text primary key, so it's not using `rowid`: ```sql CREATE TABLE [licenses] ( [key] TEXT PRIMARY KEY, [name] TEXT, [spdx_id] TEXT, [url] TEXT, [node_id] TEXT ); ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",695319258, https://github.com/simonw/sqlite-utils/issues/149#issuecomment-688460729,https://api.github.com/repos/simonw/sqlite-utils/issues/149,688460729,MDEyOklzc3VlQ29tbWVudDY4ODQ2MDcyOQ==,9599,2020-09-07T18:06:44Z,2020-09-07T18:06:44Z,OWNER,First posted on SQLite forum here but I'm pretty sure this is a bug in how `sqlite-utils` created those tables: https://sqlite.org/forum/forumpost/51aada1b45,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",695319258, https://github.com/simonw/sqlite-utils/issues/148#issuecomment-688434226,https://api.github.com/repos/simonw/sqlite-utils/issues/148,688434226,MDEyOklzc3VlQ29tbWVudDY4ODQzNDIyNg==,9599,2020-09-07T16:50:33Z,2020-09-07T16:50:33Z,OWNER,"This may be as easy as applying `textwrap.dedent()` to this: https://github.com/simonw/sqlite-utils/blob/0e62744da9a429093e3409575c1f881376b0361f/sqlite_utils/db.py#L778-L787 I could apply that to a few other queries in that code as well.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",695276328, https://github.com/dogsheep/dogsheep-beta/issues/17#issuecomment-687880459,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/17,687880459,MDEyOklzc3VlQ29tbWVudDY4Nzg4MDQ1OQ==,9599,2020-09-06T19:36:32Z,2020-09-06T19:36:32Z,MEMBER,At some point I may even want to support search types which are indexed from (and inflated from) more than one database file. I'm going to ignore that for the moment though.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",694500679, https://github.com/dogsheep/dogsheep-beta/issues/13#issuecomment-686774592,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/13,686774592,MDEyOklzc3VlQ29tbWVudDY4Njc3NDU5Mg==,9599,2020-09-03T21:30:21Z,2020-09-03T21:30:21Z,MEMBER,"This is partially supported: the custom search SQL we run doesn't escape them, but the `?_search` used to calculate facet counts does. So this is a bug.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",692386625, https://github.com/dogsheep/dogsheep-beta/issues/9#issuecomment-686767208,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/9,686767208,MDEyOklzc3VlQ29tbWVudDY4Njc2NzIwOA==,9599,2020-09-03T21:12:14Z,2020-09-03T21:12:14Z,MEMBER,Documentation: https://github.com/dogsheep/dogsheep-beta/blob/0.4/README.md#custom-results-display,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",691521965, https://github.com/dogsheep/dogsheep-beta/issues/3#issuecomment-686689612,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/3,686689612,MDEyOklzc3VlQ29tbWVudDY4NjY4OTYxMg==,9599,2020-09-03T18:44:20Z,2020-09-03T18:44:20Z,MEMBER,Facets are now displayed but selecting them doesn't work yet.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",689810340, https://github.com/dogsheep/dogsheep-beta/issues/5#issuecomment-686689366,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/5,686689366,MDEyOklzc3VlQ29tbWVudDY4NjY4OTM2Ng==,9599,2020-09-03T18:43:50Z,2020-09-03T18:43:50Z,MEMBER,No longer needed thanks to #9,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",689847361, https://github.com/dogsheep/dogsheep-beta/issues/9#issuecomment-686689122,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/9,686689122,MDEyOklzc3VlQ29tbWVudDY4NjY4OTEyMg==,9599,2020-09-03T18:43:20Z,2020-09-03T18:43:20Z,MEMBER,Needs documentation.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",691521965, https://github.com/dogsheep/dogsheep-beta/issues/9#issuecomment-686688963,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/9,686688963,MDEyOklzc3VlQ29tbWVudDY4NjY4ODk2Mw==,9599,2020-09-03T18:42:59Z,2020-09-03T18:42:59Z,MEMBER,I'm pleased with how this works now.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",691521965, https://github.com/dogsheep/dogsheep-beta/issues/11#issuecomment-686618669,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/11,686618669,MDEyOklzc3VlQ29tbWVudDY4NjYxODY2OQ==,9599,2020-09-03T16:47:34Z,2020-09-03T16:53:25Z,MEMBER,I think a `is_public` integer column which defaults to 0 would be good here.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",692125110, https://github.com/dogsheep/dogsheep-beta/issues/10#issuecomment-686238498,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/10,686238498,MDEyOklzc3VlQ29tbWVudDY4NjIzODQ5OA==,9599,2020-09-03T04:05:05Z,2020-09-03T04:05:05Z,MEMBER,Since the first two categories are `created` and `saved` this one should be called `received`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",691557547, https://github.com/dogsheep/dogsheep-beta/issues/9#issuecomment-686163754,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/9,686163754,MDEyOklzc3VlQ29tbWVudDY4NjE2Mzc1NA==,9599,2020-09-03T00:46:21Z,2020-09-03T00:46:21Z,MEMBER,"Challenge: the `dogsheep-beta.yml` configuration file that is passed to the `dogsheep-beta index` command needs to also be made available to Datasette itself, so that it can read the configuration. Let's say it can either be duplicated in the `plugins` configuration block of the `metadata.yml` OR you can do this in `metadata.yml`: ```yaml plugins: dogsheep-beta: config_file: dogsheep-beta.yml ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",691521965, https://github.com/dogsheep/dogsheep-beta/issues/9#issuecomment-686158454,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/9,686158454,MDEyOklzc3VlQ29tbWVudDY4NjE1ODQ1NA==,9599,2020-09-03T00:32:42Z,2020-09-03T00:32:42Z,MEMBER,"If this turns out to be too inefficient I could add a `display` text column to the `search_index` table which is designed to be populated with arbitrary JSON by the indexing query, which can then be used to render the template fragment.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",691521965, https://github.com/dogsheep/dogsheep-beta/issues/9#issuecomment-686154627,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/9,686154627,MDEyOklzc3VlQ29tbWVudDY4NjE1NDYyNw==,9599,2020-09-03T00:19:22Z,2020-09-03T00:19:22Z,MEMBER,If this performs well enough (100 displayed items will be 100 extra `display_sql` calls) then I'll go with this as the design for the feature.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",691521965, https://github.com/dogsheep/dogsheep-beta/issues/9#issuecomment-686154486,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/9,686154486,MDEyOklzc3VlQ29tbWVudDY4NjE1NDQ4Ng==,9599,2020-09-03T00:18:54Z,2020-09-03T00:18:54Z,MEMBER,"`display_sql` could be optional. If it's not defined, a `row` object is passed to the template which is the row that's stored in `search_index`. If `display_sql` IS defined then it's executed and the result is made available as a `display` object in addition to the `row` object.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",691521965, https://github.com/dogsheep/dogsheep-beta/issues/9#issuecomment-686153967,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/9,686153967,MDEyOklzc3VlQ29tbWVudDY4NjE1Mzk2Nw==,9599,2020-09-03T00:17:16Z,2020-09-03T00:17:55Z,MEMBER,"Maybe I can take advantage of https://sqlite.org/np1queryprob.html here - I could define a SQL query for fetching the ""display"" version of each item, and include a Jinja template fragment in the configuration as well. Maybe something like this: ```yaml photos.db: photos_with_apple_metadata: sql: |- select sha256 as key, 'Photo in ' || coalesce(place_city, 'unknown') as title, ( select group_concat(normalized_string, ' ') from labels where labels.uuid = photos_with_apple_metadata.uuid ) as search_1, date as timestamp, 1 as category from photos_with_apple_metadata display_sql: |- select sha256, place_city, date from photos_with_apple_metadata where sha256 = :key display: |-

Taken in {{ display.place_city }} on {{ display.date }}

```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",691521965, https://github.com/dogsheep/dogsheep-beta/issues/7#issuecomment-685970384,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/7,685970384,MDEyOklzc3VlQ29tbWVudDY4NTk3MDM4NA==,9599,2020-09-02T20:11:41Z,2020-09-02T20:11:59Z,MEMBER,"Default categories: - 1 = created - 2 = saved","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",691265198, https://github.com/dogsheep/dogsheep-beta/issues/7#issuecomment-685966707,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/7,685966707,MDEyOklzc3VlQ29tbWVudDY4NTk2NjcwNw==,9599,2020-09-02T20:04:08Z,2020-09-02T20:04:08Z,MEMBER,I'll make `category` a foreign key to a `categories` table so Datasette can automatically show the `name` column.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",691265198, https://github.com/dogsheep/dogsheep-beta/issues/7#issuecomment-685966361,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/7,685966361,MDEyOklzc3VlQ29tbWVudDY4NTk2NjM2MQ==,9599,2020-09-02T20:03:29Z,2020-09-02T20:03:41Z,MEMBER,"I'm going to implement the first version of this as an indexed integer `category` column which has 1 for ""about me"" and 2 for ""liked by me"" - and space for other category numerals in the future, albeit a row can only belong to one category. I'll think about a full tagging system separately.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",691265198, https://github.com/dogsheep/dogsheep-beta/issues/7#issuecomment-685965516,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/7,685965516,MDEyOklzc3VlQ29tbWVudDY4NTk2NTUxNg==,9599,2020-09-02T20:01:54Z,2020-09-02T20:01:54Z,MEMBER,"Relevant post: https://sqlite.org/forum/forumpost/9f06fedaa5 - drh says: > Indexes are one-to-one. There is one entry in the index for each row in the table. > > You are asking for an index that is many-to-one - multiple index entries for each table row. > > A Full-Text Index is basically a many-to-one index. So if all of your array entries really are words, you could probably get this to work using a Full-Text Index.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",691265198, https://github.com/dogsheep/dogsheep-beta/issues/7#issuecomment-685962280,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/7,685962280,MDEyOklzc3VlQ29tbWVudDY4NTk2MjI4MA==,9599,2020-09-02T19:55:26Z,2020-09-02T19:59:58Z,MEMBER,"Relevant: https://charlesleifer.com/blog/a-tour-of-tagging-schemas-many-to-many-bitmaps-and-more/ SQLite supports bitwise operators Binary AND (&) and Binary OR (|) - I could try those. Not sure how they interact with indexes though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",691265198, https://github.com/dogsheep/dogsheep-beta/issues/3#issuecomment-685961809,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/3,685961809,MDEyOklzc3VlQ29tbWVudDY4NTk2MTgwOQ==,9599,2020-09-02T19:54:24Z,2020-09-02T19:54:24Z,MEMBER,"This should implement search highlighting too, as seen on https://til.simonwillison.net/til/search?q=cloud ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",689810340, https://github.com/dogsheep/dogsheep-beta/issues/8#issuecomment-685960072,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/8,685960072,MDEyOklzc3VlQ29tbWVudDY4NTk2MDA3Mg==,9599,2020-09-02T19:50:47Z,2020-09-02T19:50:47Z,MEMBER,"This doesn't actually help, because the Datasette table view page doesn't then support adding the `where search_index_fts match :query` bit.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",691369691, https://github.com/dogsheep/dogsheep-beta/issues/7#issuecomment-685895540,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/7,685895540,MDEyOklzc3VlQ29tbWVudDY4NTg5NTU0MA==,9599,2020-09-02T17:46:44Z,2020-09-02T17:46:44Z,MEMBER,"Some opet questions about this: - Should I restrict to two exclusive categories here, or should I have a generic category mechanism that can be expanded to more than two? - Should an item be able to exist in more than one category? Do I want to be able to mark an indexed item as both by-me and liked-by-me for example? This question is more interesting if the number of categories is greater than two. - How should this be modeled? Single column, multiple boolean columns, JSON array, m2m against separate table? - What's the best way to make this performant","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",691265198, https://github.com/dogsheep/dogsheep-beta/issues/2#issuecomment-685121074,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/2,685121074,MDEyOklzc3VlQ29tbWVudDY4NTEyMTA3NA==,9599,2020-09-01T20:42:00Z,2020-09-01T20:42:00Z,MEMBER,Documentation at the bottom of the Usage section here: https://github.com/dogsheep/dogsheep-beta/blob/0.2/README.md#usage,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",689809225, https://github.com/dogsheep/dogsheep-beta/issues/2#issuecomment-685115519,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/2,685115519,MDEyOklzc3VlQ29tbWVudDY4NTExNTUxOQ==,9599,2020-09-01T20:31:57Z,2020-09-01T20:31:57Z,MEMBER,"Actually this doesn't work: you can't turn on stemming for specific tables, because all of the content goes into a single `search_index` table which is configured the same way. So stemming needs to be a global option.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",689809225, https://github.com/dogsheep/pocket-to-sqlite/issues/5#issuecomment-684425714,https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/5,684425714,MDEyOklzc3VlQ29tbWVudDY4NDQyNTcxNA==,9599,2020-09-01T06:18:32Z,2020-09-01T06:18:32Z,MEMBER,"Good suggestion, I'll setup a demo somewhere.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",629473827, https://github.com/dogsheep/pocket-to-sqlite/issues/3#issuecomment-684424396,https://api.github.com/repos/dogsheep/pocket-to-sqlite/issues/3,684424396,MDEyOklzc3VlQ29tbWVudDY4NDQyNDM5Ng==,9599,2020-09-01T06:17:45Z,2020-09-01T06:17:45Z,MEMBER,It looks like I could ignore the `image` column and synthesize a unique key from the data in the `images` column using `$item_id/$image_id`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",503243784, https://github.com/dogsheep/dogsheep-beta/issues/4#issuecomment-684395444,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/4,684395444,MDEyOklzc3VlQ29tbWVudDY4NDM5NTQ0NA==,9599,2020-09-01T06:00:03Z,2020-09-01T06:00:03Z,MEMBER,I ran `sqlite-utils optimize beta.db` against my test DB and the size reduced from 183M to 176M - and a 450ms search ran in 359ms. So not a huge improvement but still worthwhile.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",689839399, https://github.com/dogsheep/dogsheep-beta/issues/3#issuecomment-684250044,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/3,684250044,MDEyOklzc3VlQ29tbWVudDY4NDI1MDA0NA==,9599,2020-09-01T05:01:09Z,2020-09-01T05:01:23Z,MEMBER,Maybe this starts out as a custom templated canned query.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",689810340, https://github.com/simonw/sqlite-utils/issues/147#issuecomment-683528149,https://api.github.com/repos/simonw/sqlite-utils/issues/147,683528149,MDEyOklzc3VlQ29tbWVudDY4MzUyODE0OQ==,9599,2020-08-31T03:17:26Z,2020-08-31T03:17:26Z,OWNER,"+1 to making this something that users can customize. An optional argument to the `Database` constructor would be a neat way to do this. I think there's a terrifying way that we could find this value... we could perform a binary search for it! Open up a memory connection and try running different bulk inserts against it and catch the exceptions - then adjust and try again. My hunch is that we could perform just 2 or 3 probes (maybe against carefully selected values) to find the highest value that works. If this process took less than a few ms to run I'd be happy to do it automatically when the class is instantiated (and let users disable that automatic proving by setting a value using the constructor argument).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",688670158, https://github.com/simonw/datasette/issues/948#issuecomment-683448569,https://api.github.com/repos/simonw/datasette/issues/948,683448569,MDEyOklzc3VlQ29tbWVudDY4MzQ0ODU2OQ==,9599,2020-08-30T17:39:09Z,2020-08-30T18:34:34Z,OWNER,"So the steps needed are: - Download and extract latest CodeMirror zip file - Rename `lib/codemirror.js` to `codemirror-5.57.0.js` - Rename `lib/codemirror.css` to `codemirror-5.57.0.css` - Rename `mode/sql/sql.js` to `codemirror-5.57.0-sql.js` - Edit both JS files to make the top comment a `/* */` block - Minify JavaScript files like this: - `npx uglify-js codemirror-5.57.0.js -o codemirror-5.57.0.min.js --comments '/LICENSE/'` - `npx uglify-js codemirror-5.57.0-sql.js -o codemirror-5.57.0-sql.min.js --comments '/LICENSE/'` - Check that the LICENSE comment did indeed survive minification - Minify CSS file like this: - `npx clean-css-cli codemirror-5.57.0.css -o codemirror-5.57.0.min.css` - Edit the `_codemirror.html` template to reference the new files - `git rm` the old files, `git add` the new files","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",684925907, https://github.com/simonw/datasette/issues/948#issuecomment-683452613,https://api.github.com/repos/simonw/datasette/issues/948,683452613,MDEyOklzc3VlQ29tbWVudDY4MzQ1MjYxMw==,9599,2020-08-30T18:16:28Z,2020-08-30T18:16:28Z,OWNER,I added documentation on how to upgrade CodeMirror for the future here: https://docs.datasette.io/en/latest/contributing.html#upgrading-codemirror,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",684925907, https://github.com/simonw/datasette/issues/655#issuecomment-683449837,https://api.github.com/repos/simonw/datasette/issues/655,683449837,MDEyOklzc3VlQ29tbWVudDY4MzQ0OTgzNw==,9599,2020-08-30T17:51:38Z,2020-08-30T17:51:38Z,OWNER,I think was fixed by #948,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",542553350, https://github.com/simonw/datasette/issues/948#issuecomment-683449804,https://api.github.com/repos/simonw/datasette/issues/948,683449804,MDEyOklzc3VlQ29tbWVudDY4MzQ0OTgwNA==,9599,2020-08-30T17:51:18Z,2020-08-30T17:51:18Z,OWNER,Copy and paste on mobile safari seems to work now. #655 ,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",684925907, https://github.com/simonw/datasette/issues/948#issuecomment-683448635,https://api.github.com/repos/simonw/datasette/issues/948,683448635,MDEyOklzc3VlQ29tbWVudDY4MzQ0ODYzNQ==,9599,2020-08-30T17:39:54Z,2020-08-30T17:39:54Z,OWNER,I'll wait for this to deploy to https://latest.datasette.io/ and then test it in various desktop and mobile browsers.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",684925907, https://github.com/simonw/datasette/issues/948#issuecomment-683445704,https://api.github.com/repos/simonw/datasette/issues/948,683445704,MDEyOklzc3VlQ29tbWVudDY4MzQ0NTcwNA==,9599,2020-08-30T17:11:58Z,2020-08-30T17:33:30Z,OWNER,"One catch: this stripped the license information from the top of the JS. I fixed this by editing the license to be a single `/* ... */` block comment instead of multiple `//` lines and running this: npx uglify-js codemirror-5.57.0.js -o codemirror-5.57.0.min.js --comments '/LICENSE/' ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",684925907, https://github.com/simonw/datasette/issues/948#issuecomment-683445114,https://api.github.com/repos/simonw/datasette/issues/948,683445114,MDEyOklzc3VlQ29tbWVudDY4MzQ0NTExNA==,9599,2020-08-30T17:06:39Z,2020-08-30T17:06:39Z,OWNER,"Minifying using `npx`: ``` npx uglify-js codemirror-5.57.0.js -o codemirror-5.57.0.min.js npx uglify-js codemirror-5.57.0-sql.js -o codemirror-5.57.0-sql.min.js npx clean-css-cli codemirror-5.57.0.css -o codemirror-5.57.0.min.css ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",684925907, https://github.com/simonw/datasette/issues/957#issuecomment-683357092,https://api.github.com/repos/simonw/datasette/issues/957,683357092,MDEyOklzc3VlQ29tbWVudDY4MzM1NzA5Mg==,9599,2020-08-30T00:15:51Z,2020-08-30T00:16:02Z,OWNER,"Weirdly even removing this single `datasette` import from `utils/asgi.py` didn't fix the circular import: https://github.com/simonw/datasette/blob/44cf424a94a85b74552075272660bb96a7432661/datasette/utils/asgi.py#L1-L3","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",688622148, https://github.com/simonw/datasette/issues/957#issuecomment-683356440,https://api.github.com/repos/simonw/datasette/issues/957,683356440,MDEyOklzc3VlQ29tbWVudDY4MzM1NjQ0MA==,9599,2020-08-30T00:08:18Z,2020-08-30T00:10:26Z,OWNER,"Annoyingly this seems to be the line that causes the circular import: ```python from .utils.asgi import Forbidden, NotFound, Response ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",688622148, https://github.com/simonw/datasette/issues/957#issuecomment-683355993,https://api.github.com/repos/simonw/datasette/issues/957,683355993,MDEyOklzc3VlQ29tbWVudDY4MzM1NTk5Mw==,9599,2020-08-30T00:02:11Z,2020-08-30T00:04:18Z,OWNER,"I tried doing this and got this error: ``` (datasette) datasette % pytest ==================================================================== test session starts ===================================================================== platform darwin -- Python 3.8.5, pytest-6.0.1, py-1.9.0, pluggy-0.13.1 rootdir: /Users/simon/Dropbox/Development/datasette, configfile: pytest.ini plugins: asyncio-0.14.0, timeout-1.4.2 collected 1 item / 23 errors =========================================================================== ERRORS =========================================================================== _____________________________________________________________ ERROR collecting tests/test_api.py _____________________________________________________________ ImportError while importing test module '/Users/simon/Dropbox/Development/datasette/tests/test_api.py'. Hint: make sure your test modules/packages have valid Python names. Traceback: /usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/importlib/__init__.py:127: in import_module return _bootstrap._gcd_import(name[level:], package, level) tests/test_api.py:1: in from datasette.plugins import DEFAULT_PLUGINS datasette/__init__.py:2: in from .app import Datasette datasette/app.py:30: in from .views.base import DatasetteError, ureg datasette/views/base.py:12: in from datasette.plugins import pm datasette/plugins.py:26: in mod = importlib.import_module(plugin) /usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/importlib/__init__.py:127: in import_module return _bootstrap._gcd_import(name[level:], package, level) datasette/publish/heroku.py:2: in from datasette import hookimpl E ImportError: cannot import name 'hookimpl' from partially initialized module 'datasette' (most likely due to a circular import) (/Users/simon/Dropbox/Development/datasette/datasette/__init__.py) ``` That's with `datasette/__init__.py` looking like this: ```python from datasette.version import __version_info__, __version__ # noqa from .app import Datasette from .utils.asgi import Forbidden, NotFound, Response from .utils import actor_matches_allow, QueryInterrupted from .hookspecs import hookimpl # noqa from .hookspecs import hookspec # noqa __all__ = [ ""actor_matches_allow"", ""hookimpl"", ""hookspec"", ""QueryInterrupted"", ""Forbidden"", ""NotFound"", ""Response"", ""Datasette"", ] ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",688622148, https://github.com/simonw/datasette/issues/957#issuecomment-683355598,https://api.github.com/repos/simonw/datasette/issues/957,683355598,MDEyOklzc3VlQ29tbWVudDY4MzM1NTU5OA==,9599,2020-08-29T23:55:10Z,2020-08-29T23:55:34Z,OWNER,"Of these I think I'm going to promote the following to being importable directly `from datasette`: - `from datasette.app import Datasette` - `from datasette.utils import QueryInterrupted` - `from datasette.utils.asgi import Response, Forbidden, NotFound` - `from datasette.utils import actor_matches_allow` All of the rest are infrequently used enough (or clearly named enough) that I'm happy to leave them as-is.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",688622148, https://github.com/simonw/datasette/issues/957#issuecomment-683355508,https://api.github.com/repos/simonw/datasette/issues/957,683355508,MDEyOklzc3VlQ29tbWVudDY4MzM1NTUwOA==,9599,2020-08-29T23:54:01Z,2020-08-29T23:54:01Z,OWNER,"Reviewing https://github.com/search?q=user%3Asimonw+%22from+datasette%22&type=Code I spotted these others: ```python # Various: from datasette.utils import path_with_replaced_args from datasette.plugins import pm from datasette.utils import QueryInterrupted from datasette.utils.asgi import Response, Forbidden, NotFound # datasette-publish-vercel: from datasette.publish.common import ( add_common_publish_arguments_and_options, fail_if_publish_binary_not_installed ) from datasette.utils import temporary_docker_directory # datasette-insert from datasette.utils import actor_matches_allow, sqlite3 # obsolete: russian-ira-facebook-ads-datasette from datasette.utils import TableFilter # simonw/museums from datasette.utils.asgi import asgi_send # datasette-media from datasette.utils.asgi import Response, asgi_send_file # datasette/tests/plugins/my_plugin.py from datasette.facets import Facet # datasette-graphql from datasette.views.table import TableView ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",688622148, https://github.com/simonw/datasette/issues/956#issuecomment-683214102,https://api.github.com/repos/simonw/datasette/issues/956,683214102,MDEyOklzc3VlQ29tbWVudDY4MzIxNDEwMg==,9599,2020-08-29T01:32:21Z,2020-08-29T01:32:21Z,OWNER,Maybe the bug here is the double colon?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",688427751, https://github.com/simonw/datasette/issues/956#issuecomment-683213973,https://api.github.com/repos/simonw/datasette/issues/956,683213973,MDEyOklzc3VlQ29tbWVudDY4MzIxMzk3Mw==,9599,2020-08-29T01:31:39Z,2020-08-29T01:31:39Z,OWNER,"Here's how the old Travis mechanism worked: https://github.com/simonw/datasette/blob/52eabb019d4051084b21524bd0fd9c2731126985/.travis.yml#L41-L47 So I was assuming that the eqivalent of `$REPO:$TRAVIS_TAG` in GitHub Actions is `$REPO::${GITHUB_REF#refs/tags/}`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",688427751, https://github.com/simonw/datasette/issues/956#issuecomment-683212960,https://api.github.com/repos/simonw/datasette/issues/956,683212960,MDEyOklzc3VlQ29tbWVudDY4MzIxMjk2MA==,9599,2020-08-29T01:25:34Z,2020-08-29T01:25:34Z,OWNER,So I guess this bit is wrong: https://github.com/simonw/datasette/blob/c36e287d71d68ecb2a45e9808eede15f19f931fb/.github/workflows/publish.yml#L71-L73,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",688427751, https://github.com/simonw/datasette/issues/956#issuecomment-683212421,https://api.github.com/repos/simonw/datasette/issues/956,683212421,MDEyOklzc3VlQ29tbWVudDY4MzIxMjQyMQ==,9599,2020-08-29T01:22:23Z,2020-08-29T01:22:23Z,OWNER,"Here's the error message again: > invalid argument `""***/datasette::0.49a0""` for `""-t, --tag""` flag: invalid reference format","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",688427751, https://github.com/simonw/datasette/issues/956#issuecomment-683212246,https://api.github.com/repos/simonw/datasette/issues/956,683212246,MDEyOklzc3VlQ29tbWVudDY4MzIxMjI0Ng==,9599,2020-08-29T01:21:26Z,2020-08-29T01:21:26Z,OWNER,I added this but I have no idea if I got it right or not: https://github.com/simonw/datasette/blob/c36e287d71d68ecb2a45e9808eede15f19f931fb/.github/workflows/publish.yml#L58-L63,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",688427751, https://github.com/simonw/datasette/issues/955#issuecomment-683189334,https://api.github.com/repos/simonw/datasette/issues/955,683189334,MDEyOklzc3VlQ29tbWVudDY4MzE4OTMzNA==,9599,2020-08-28T23:30:48Z,2020-08-28T23:30:48Z,OWNER,Also https://github.com/simonw/datasette-copyable,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",687711713, https://github.com/simonw/datasette/issues/955#issuecomment-683185861,https://api.github.com/repos/simonw/datasette/issues/955,683185861,MDEyOklzc3VlQ29tbWVudDY4MzE4NTg2MQ==,9599,2020-08-28T23:17:09Z,2020-08-28T23:17:09Z,OWNER,I released 0.49a0 which means I can update the main branches of those two plugins - I'll push a release of them once 0.49 is fully shipped.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",687711713, https://github.com/simonw/sqlite-utils/issues/144#issuecomment-683180581,https://api.github.com/repos/simonw/sqlite-utils/issues/144,683180581,MDEyOklzc3VlQ29tbWVudDY4MzE4MDU4MQ==,9599,2020-08-28T22:57:04Z,2020-08-28T22:57:04Z,OWNER,"That worked! https://github.com/simonw/sqlite-utils/runs/1043640785?check_suite_focus=true ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",688395275, https://github.com/simonw/sqlite-utils/issues/144#issuecomment-683179678,https://api.github.com/repos/simonw/sqlite-utils/issues/144,683179678,MDEyOklzc3VlQ29tbWVudDY4MzE3OTY3OA==,9599,2020-08-28T22:53:17Z,2020-08-28T22:53:17Z,OWNER,I'm going to try doing this as a GitHub Actions test matrix.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",688395275, https://github.com/simonw/sqlite-utils/issues/139#issuecomment-683178570,https://api.github.com/repos/simonw/sqlite-utils/issues/139,683178570,MDEyOklzc3VlQ29tbWVudDY4MzE3ODU3MA==,9599,2020-08-28T22:48:51Z,2020-08-28T22:48:51Z,OWNER,"Thanks @simonwiles, this is now released in 2.16.1: https://sqlite-utils.readthedocs.io/en/stable/changelog.html","{""total_count"": 2, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 1, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",686978131, https://github.com/simonw/sqlite-utils/issues/143#issuecomment-683175491,https://api.github.com/repos/simonw/sqlite-utils/issues/143,683175491,MDEyOklzc3VlQ29tbWVudDY4MzE3NTQ5MQ==,9599,2020-08-28T22:37:15Z,2020-08-28T22:37:15Z,OWNER,"I'm going to start running black exclusively in the GitHub Actions workflow, rather than having it run by the unit tests themselves.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",688389933, https://github.com/simonw/sqlite-utils/pull/142#issuecomment-683173375,https://api.github.com/repos/simonw/sqlite-utils/issues/142,683173375,MDEyOklzc3VlQ29tbWVudDY4MzE3MzM3NQ==,9599,2020-08-28T22:29:02Z,2020-08-28T22:29:02Z,OWNER,Yeah I think that failure is actually because there's a brand new release of Black out and it subtly changes some of the formatting rules. I'll merge this and then run Black against the entire codebase.,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",688386219, https://github.com/simonw/sqlite-utils/pull/142#issuecomment-683172829,https://api.github.com/repos/simonw/sqlite-utils/issues/142,683172829,MDEyOklzc3VlQ29tbWVudDY4MzE3MjgyOQ==,9599,2020-08-28T22:27:05Z,2020-08-28T22:27:05Z,OWNER,"Looks like it failed the ""black"" formatting test - possibly because there's a new release if black out. I'm going to merge despite that failure.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",688386219, https://github.com/simonw/sqlite-utils/pull/142#issuecomment-683172082,https://api.github.com/repos/simonw/sqlite-utils/issues/142,683172082,MDEyOklzc3VlQ29tbWVudDY4MzE3MjA4Mg==,9599,2020-08-28T22:24:25Z,2020-08-28T22:24:25Z,OWNER,Thanks very much!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",688386219, https://github.com/simonw/sqlite-utils/issues/119#issuecomment-683146200,https://api.github.com/repos/simonw/sqlite-utils/issues/119,683146200,MDEyOklzc3VlQ29tbWVudDY4MzE0NjIwMA==,9599,2020-08-28T21:05:37Z,2020-08-28T21:05:37Z,OWNER,Maybe use `transform_table()` in #114 for this? Would be less efficient as it would copy the whole table but it would reduce library complexity a bit.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",652700770, https://github.com/simonw/sqlite-utils/issues/139#issuecomment-682771226,https://api.github.com/repos/simonw/sqlite-utils/issues/139,682771226,MDEyOklzc3VlQ29tbWVudDY4Mjc3MTIyNg==,9599,2020-08-28T15:57:42Z,2020-08-28T15:57:42Z,OWNER,"That pull request should update this section of the documentation too: > If you have more than one record to insert, the insert_all() method is a much more efficient way of inserting them. Just like insert() it will automatically detect the columns that should be created, but it will inspect the first batch of 100 items to help decide what those column types should be. https://github.com/simonw/sqlite-utils/blob/ea87c2b943fdd162c42a900ac0aea5ecc2f4b9d9/docs/python-api.rst#L393","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",686978131, https://github.com/simonw/sqlite-utils/issues/139#issuecomment-682762911,https://api.github.com/repos/simonw/sqlite-utils/issues/139,682762911,MDEyOklzc3VlQ29tbWVudDY4Mjc2MjkxMQ==,9599,2020-08-28T15:54:57Z,2020-08-28T15:55:20Z,OWNER,"Here's a suggested test update: ```diff diff --git a/sqlite_utils/db.py b/sqlite_utils/db.py index a8791c3..12fa2f2 100644 --- a/sqlite_utils/db.py +++ b/sqlite_utils/db.py @@ -1074,6 +1074,13 @@ class Table(Queryable): all_columns = list(sorted(all_columns)) if hash_id: all_columns.insert(0, hash_id) + else: + all_columns += [ + column + for record in chunk + for column in record + if column not in all_columns + ] validate_column_names(all_columns) first = False # values is the list of insert data that is passed to the diff --git a/tests/test_create.py b/tests/test_create.py index a84eb8d..3a7fafc 100644 --- a/tests/test_create.py +++ b/tests/test_create.py @@ -707,13 +707,15 @@ def test_insert_thousands_using_generator(fresh_db): assert 10000 == fresh_db[""test""].count -def test_insert_thousands_ignores_extra_columns_after_first_100(fresh_db): +def test_insert_thousands_adds_extra_columns_after_first_100(fresh_db): + # https://github.com/simonw/sqlite-utils/issues/139 fresh_db[""test""].insert_all( [{""i"": i, ""word"": ""word_{}"".format(i)} for i in range(100)] - + [{""i"": 101, ""extra"": ""This extra column should cause an exception""}] + + [{""i"": 101, ""extra"": ""Should trigger ALTER""}], + alter=True, ) rows = fresh_db.execute_returning_dicts(""select * from test where i = 101"") - assert [{""i"": 101, ""word"": None}] == rows + assert [{""i"": 101, ""word"": None, ""extra"": ""Should trigger ALTER""}] == rows def test_insert_ignore(fresh_db): ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",686978131, https://github.com/simonw/datasette/issues/954#issuecomment-682312736,https://api.github.com/repos/simonw/datasette/issues/954,682312736,MDEyOklzc3VlQ29tbWVudDY4MjMxMjczNg==,9599,2020-08-28T04:05:01Z,2020-08-28T04:05:10Z,OWNER,> It can also return a dictionary with the following keys. This format is **deprecated** as-of Datasette 0.49 and will be removed by Datasette 1.0.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",687694947, https://github.com/simonw/datasette/issues/953#issuecomment-682312494,https://api.github.com/repos/simonw/datasette/issues/953,682312494,MDEyOklzc3VlQ29tbWVudDY4MjMxMjQ5NA==,9599,2020-08-28T04:03:56Z,2020-08-28T04:03:56Z,OWNER,"Documentation says that the old dictionary mechanism will be deprecated by 1.0: https://github.com/simonw/datasette/blob/799ecae94824640bdff21f86997f69844048d5c3/docs/plugin_hooks.rst#L460","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",687681018, https://github.com/simonw/sqlite-utils/issues/139#issuecomment-682285212,https://api.github.com/repos/simonw/sqlite-utils/issues/139,682285212,MDEyOklzc3VlQ29tbWVudDY4MjI4NTIxMg==,9599,2020-08-28T02:12:51Z,2020-08-28T02:12:51Z,OWNER,"I'd be happy to accept a PR for this, provided it included updated unit tests that illustrate it working. I think this is a really good improvement.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",686978131, https://github.com/simonw/sqlite-utils/issues/139#issuecomment-682284908,https://api.github.com/repos/simonw/sqlite-utils/issues/139,682284908,MDEyOklzc3VlQ29tbWVudDY4MjI4NDkwOA==,9599,2020-08-28T02:11:40Z,2020-08-28T02:11:40Z,OWNER,"This is deliberate behaviour, but I'm not at all attached to it - you're right in pointing out that it's actually pretty unexpected. I'd be happy to change this behaviour so if you pass `alter=True` and then use `.insert_all()` on more than 100 rows it works as you would expect, instead of silently ignoring new columns past the first 100 rows. I don't expect that anyone would be depending on the current behaviour (ignore new columns after the first 100) such that this should be considered a backwards incompatible change.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",686978131, https://github.com/simonw/datasette/issues/950#issuecomment-680374196,https://api.github.com/repos/simonw/datasette/issues/950,680374196,MDEyOklzc3VlQ29tbWVudDY4MDM3NDE5Ng==,9599,2020-08-26T00:43:50Z,2020-08-26T00:43:50Z,OWNER,"The problem with the term ""private"" is that it could be confused with the concept of databases that aren't visible to the public due to the permissions system - the ones that are displayed with the padlock icon e.g. on https://datasette-auth-passwords-demo.datasette.io/ So I think ""secret"" is a better term for these.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",685806511, https://github.com/simonw/datasette/issues/950#issuecomment-680264202,https://api.github.com/repos/simonw/datasette/issues/950,680264202,MDEyOklzc3VlQ29tbWVudDY4MDI2NDIwMg==,9599,2020-08-25T20:53:13Z,2020-08-25T20:53:13Z,OWNER,Forcing people to spell out `datasette github.db --private private.db` isn't terrible though.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",685806511, https://github.com/simonw/datasette/issues/950#issuecomment-680263999,https://api.github.com/repos/simonw/datasette/issues/950,680263999,MDEyOklzc3VlQ29tbWVudDY4MDI2Mzk5OQ==,9599,2020-08-25T20:52:47Z,2020-08-25T20:52:47Z,OWNER,"Naming challenge: secret databases or private databases? I prefer private. But `datasette -p` is already taken by `--port`. `datasette -s` is currently available.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",685806511, https://github.com/simonw/datasette/issues/950#issuecomment-680263427,https://api.github.com/repos/simonw/datasette/issues/950,680263427,MDEyOklzc3VlQ29tbWVudDY4MDI2MzQyNw==,9599,2020-08-25T20:51:30Z,2020-08-25T20:52:13Z,OWNER,"`datasette-graphql` currently dispatches requests through the `TableView` class, so if that couldn't access private databases then it would not be able to either. See also the concept for `datasette.get(...)` as an internal API in #943 - that might need to have a mechanism for also being able to query private databases, maybe `datasette.get(path, allow_private=True)`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",685806511, https://github.com/simonw/datasette/issues/949#issuecomment-679367931,https://api.github.com/repos/simonw/datasette/issues/949,679367931,MDEyOklzc3VlQ29tbWVudDY3OTM2NzkzMQ==,9599,2020-08-24T21:09:50Z,2020-08-24T21:09:50Z,OWNER,"I'm attracted to this because of how good GraphiQL is for auto-completing queries. But I realize there's a problem here: GraphQL is designed to be autocomplete-friendly, but SQL is not. If you type `select ` and it doesn't know what's going in the `from` clause it can't give you good column autocomplete, for example.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",684961449, https://github.com/simonw/datasette/issues/949#issuecomment-679363710,https://api.github.com/repos/simonw/datasette/issues/949,679363710,MDEyOklzc3VlQ29tbWVudDY3OTM2MzcxMA==,9599,2020-08-24T21:00:43Z,2020-08-24T21:00:43Z,OWNER,"I think this requires three extra files from https://github.com/codemirror/CodeMirror/tree/5.57.0/addon/hint - `show-hint.css` - `show-hint.js` - `sql-hint.js` ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",684961449, https://github.com/simonw/datasette/issues/948#issuecomment-679355426,https://api.github.com/repos/simonw/datasette/issues/948,679355426,MDEyOklzc3VlQ29tbWVudDY3OTM1NTQyNg==,9599,2020-08-24T20:43:07Z,2020-08-24T20:43:07Z,OWNER,"It would also be interesting to try out the SQL hint mode, which can autocomplete against tables and columns. This demo shows how to configure that: https://codemirror.net/mode/sql/","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",684925907, https://github.com/simonw/datasette/issues/948#issuecomment-679333717,https://api.github.com/repos/simonw/datasette/issues/948,679333717,MDEyOklzc3VlQ29tbWVudDY3OTMzMzcxNw==,9599,2020-08-24T19:55:59Z,2020-08-24T19:55:59Z,OWNER,CodeMirror 6 is in pre-release at the moment and is a complete rewrite. I'll stick with the 5.x series for now. https://github.com/codemirror/codemirror.next/,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",684925907, https://github.com/simonw/sqlite-utils/issues/138#issuecomment-678732667,https://api.github.com/repos/simonw/sqlite-utils/issues/138,678732667,MDEyOklzc3VlQ29tbWVudDY3ODczMjY2Nw==,9599,2020-08-23T05:46:10Z,2020-08-23T05:46:10Z,OWNER,"Actually the `TEXT` column thing wasn't a `sqlite-utils` issue, it was unique to how `shapefile-to-spatialite` was creating the table when using the SpatiaLite extension.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",684118950, https://github.com/simonw/sqlite-utils/issues/136#issuecomment-678508056,https://api.github.com/repos/simonw/sqlite-utils/issues/136,678508056,MDEyOklzc3VlQ29tbWVudDY3ODUwODA1Ng==,9599,2020-08-21T21:13:41Z,2020-08-21T21:13:41Z,OWNER,"The `--spatialite` option should be available for other useful commands too, refs #137.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",683812642, https://github.com/simonw/sqlite-utils/issues/137#issuecomment-678507502,https://api.github.com/repos/simonw/sqlite-utils/issues/137,678507502,MDEyOklzc3VlQ29tbWVudDY3ODUwNzUwMg==,9599,2020-08-21T21:13:19Z,2020-08-21T21:13:19Z,OWNER,Adding `--spatialite` too would be great for usability: #136,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",683830416, https://github.com/simonw/sqlite-utils/issues/134#issuecomment-678497497,https://api.github.com/repos/simonw/sqlite-utils/issues/134,678497497,MDEyOklzc3VlQ29tbWVudDY3ODQ5NzQ5Nw==,9599,2020-08-21T21:06:26Z,2020-08-21T21:06:26Z,OWNER,"Ended up needing two skipIfs: https://github.com/simonw/sqlite-utils/blob/7e9aad7e1c09d1cf80d0b4d17d6157212a4b857d/tests/test_cli.py#L888-L893","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",683804172, https://github.com/simonw/sqlite-utils/issues/136#issuecomment-678480969,https://api.github.com/repos/simonw/sqlite-utils/issues/136,678480969,MDEyOklzc3VlQ29tbWVudDY3ODQ4MDk2OQ==,9599,2020-08-21T20:33:45Z,2020-08-21T20:33:45Z,OWNER,"I think this should initialize SpatiaLite against the current database if it has not been initialized already. Relevant code: https://github.com/simonw/shapefile-to-sqlite/blob/e754d0747ca2facf9a7433e2d5d15a6a37a9cf6e/shapefile_to_sqlite/utils.py#L112-L126 ```python def init_spatialite(db, lib): db.conn.enable_load_extension(True) db.conn.load_extension(lib) # Initialize SpatiaLite if not yet initialized if ""spatial_ref_sys"" in db.table_names(): return db.conn.execute(""select InitSpatialMetadata(1)"") def ensure_table_has_geometry(db, table, table_srid): if ""geometry"" not in db[table].columns_dict: db.conn.execute( ""SELECT AddGeometryColumn(?, 'geometry', ?, 'GEOMETRY', 2);"", [table, table_srid], ) ``` Not sure if I should add a utility function or CLI command for that `ensure_table_has_geometry` bit.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",683812642, https://github.com/simonw/sqlite-utils/issues/135#issuecomment-678479741,https://api.github.com/repos/simonw/sqlite-utils/issues/135,678479741,MDEyOklzc3VlQ29tbWVudDY3ODQ3OTc0MQ==,9599,2020-08-21T20:30:37Z,2020-08-21T20:30:37Z,OWNER,Docs: https://github.com/simonw/sqlite-utils/blob/bf4c6b7c82fab6b2400e48424f8dac1ae2f0a2dc/docs/python-api.rst#finding-spatialite,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",683805434, https://github.com/simonw/sqlite-utils/issues/135#issuecomment-678476842,https://api.github.com/repos/simonw/sqlite-utils/issues/135,678476842,MDEyOklzc3VlQ29tbWVudDY3ODQ3Njg0Mg==,9599,2020-08-21T20:23:13Z,2020-08-21T20:23:13Z,OWNER,I'm going to start with just the first two - I'm not convinced I understand the `.so.5` variants.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",683805434, https://github.com/simonw/sqlite-utils/issues/134#issuecomment-678476338,https://api.github.com/repos/simonw/sqlite-utils/issues/134,678476338,MDEyOklzc3VlQ29tbWVudDY3ODQ3NjMzOA==,9599,2020-08-21T20:22:02Z,2020-08-21T20:22:02Z,OWNER,I think that adds it as `/usr/lib/x86_64-linux-gnu/mod_spatialite.so`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",683804172, https://github.com/simonw/sqlite-utils/issues/135#issuecomment-678475578,https://api.github.com/repos/simonw/sqlite-utils/issues/135,678475578,MDEyOklzc3VlQ29tbWVudDY3ODQ3NTU3OA==,9599,2020-08-21T20:20:05Z,2020-08-21T20:20:05Z,OWNER,"https://github.com/simonw/cryptozoology/blob/2ad69168f3b78ebd90a2cbeea8136c9115e2a9b7/build_cryptids_database.py#L16-L22 ```python try_these = ( ""mod_spatialite"", ""/usr/local/lib/mod_spatialite.dylib"", ""/usr/lib/x86_64-linux-gnu/mod_spatialite.so"", ""/usr/lib/x86_64-linux-gnu/libspatialite.so.5"", ""/usr/lib/x86_64-linux-gnu/libspatialite.so.7"", ) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",683805434, https://github.com/simonw/sqlite-utils/issues/134#issuecomment-678474928,https://api.github.com/repos/simonw/sqlite-utils/issues/134,678474928,MDEyOklzc3VlQ29tbWVudDY3ODQ3NDkyOA==,9599,2020-08-21T20:18:33Z,2020-08-21T20:18:33Z,OWNER,"This should get me SpatiaLite in the GitHub Actions Ubuntu: ``` apt install libsqlite3-mod-spatialite ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",683804172, https://github.com/simonw/sqlite-utils/issues/134#issuecomment-678474018,https://api.github.com/repos/simonw/sqlite-utils/issues/134,678474018,MDEyOklzc3VlQ29tbWVudDY3ODQ3NDAxOA==,9599,2020-08-21T20:16:20Z,2020-08-21T20:16:20Z,OWNER,"Trickiest part of this is how to write a test for it. I'll do a `pytest.skipIf` that only executes the test if SpatiaLite is available.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",683804172, https://github.com/simonw/datasette/issues/945#issuecomment-676556377,https://api.github.com/repos/simonw/datasette/issues/945,676556377,MDEyOklzc3VlQ29tbWVudDY3NjU1NjM3Nw==,9599,2020-08-19T17:21:16Z,2020-08-19T17:21:16Z,OWNER,Documented here: https://docs.datasette.io/en/latest/plugins.html#installing-plugins,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",682005535, https://github.com/simonw/datasette/issues/943#issuecomment-675889865,https://api.github.com/repos/simonw/datasette/issues/943,675889865,MDEyOklzc3VlQ29tbWVudDY3NTg4OTg2NQ==,9599,2020-08-19T06:57:00Z,2020-08-19T06:57:00Z,OWNER,Maybe `.get` vs `.get_html`?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681375466, https://github.com/simonw/datasette/issues/943#issuecomment-675889551,https://api.github.com/repos/simonw/datasette/issues/943,675889551,MDEyOklzc3VlQ29tbWVudDY3NTg4OTU1MQ==,9599,2020-08-19T06:56:06Z,2020-08-19T06:56:17Z,OWNER,"I'm leaning towards defaulting to JSON as the requested format - you can pass `format=""html""` if you want HTML. But weird that it's different from the web UI.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681375466, https://github.com/simonw/datasette/issues/943#issuecomment-675884980,https://api.github.com/repos/simonw/datasette/issues/943,675884980,MDEyOklzc3VlQ29tbWVudDY3NTg4NDk4MA==,9599,2020-08-19T06:44:26Z,2020-08-19T06:44:26Z,OWNER,"Need to decide what to do about JSON responses. When called from a template it's likely the intent will be to further loop through the JSON data returned. It would be annoying to have to run `json.loads` here. Maybe a `.get_json()` method then? Or even return a response that has `.json()` and `.text` similar to `httpx` - or just return an `httpx` response.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681375466, https://github.com/simonw/datasette/issues/944#issuecomment-675830678,https://api.github.com/repos/simonw/datasette/issues/944,675830678,MDEyOklzc3VlQ29tbWVudDY3NTgzMDY3OA==,9599,2020-08-19T03:30:10Z,2020-08-19T03:30:10Z,OWNER,"These templates will need a way to raise a 404 - so that if the template itself is deciding if the page exists (for example using `datasette-template-sql` or the proposed `datasette.get()` method from #943 or the `graphql()` template function in https://github.com/simonw/datasette-graphql/issues/50) it can return a regular 404 page. This can imitate the `custom_redirect()` function from https://docs.datasette.io/en/stable/custom_templates.html#custom-redirects: ```html+jinja {{ custom_redirect(""https://github.com/simonw/datasette"", 301) }} ``` It could be as simple as this: ``` {{ raise_404(""Museum not found"") }} ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681516976, https://github.com/simonw/datasette/issues/944#issuecomment-675829942,https://api.github.com/repos/simonw/datasette/issues/944,675829942,MDEyOklzc3VlQ29tbWVudDY3NTgyOTk0Mg==,9599,2020-08-19T03:27:25Z,2020-08-19T03:27:25Z,OWNER,"I created a template file called `templates/pages/museums/{slug}.html` and used the debugger to see if Jinja could see it. This worked: ``` (Pdb) self.ds.jinja_env.list_templates() ['500.html', '_codemirror.html', '_codemirror_foot.html', '_description_source_license.html', '_footer.html', '_table.html', 'allow_debug.html', 'base.html', 'database.html', 'default:500.html', 'default:_codemirror.html', 'default:_codemirror_foot.html', 'default:_description_source_license.html', 'default:_footer.html', 'default:_table.html', 'default:allow_debug.html', 'default:base.html', 'default:database.html', 'default:index.html', 'default:logout.html', 'default:messages_debug.html', 'default:patterns.html', 'default:permissions_debug.html', 'default:query.html', 'default:row.html', 'default:show_json.html', 'default:table.html', 'forbidden.html', 'index.html', 'logout.html', 'messages_debug.html', 'pages/about.html', 'pages/museums/{slug}.html', 'patterns.html', 'permissions_debug.html', 'query.html', 'row.html', 'show_json.html', 'table.html'] ``` The `pages/museums/{slug}.html` template is in that list. Here's the implementation of that `list_templates()` method - it does some filesystem walking so it may be a bit expensive to run it on every request: https://github.com/pallets/jinja/blob/ca8b0b0287e320fe1f4a74f36910ef7ae3303d99/src/jinja2/loaders.py#L197-L212 But caching it would be pretty easy - either until the server is restarted or as an in-memory cache for a few seconds.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681516976, https://github.com/simonw/datasette/issues/943#issuecomment-675788203,https://api.github.com/repos/simonw/datasette/issues/943,675788203,MDEyOklzc3VlQ29tbWVudDY3NTc4ODIwMw==,9599,2020-08-19T00:46:08Z,2020-08-19T00:46:23Z,OWNER,Also fun: the inevitable plugin that exposes this to the template language - so Datasette templates can stitch together data from multiple other internal API calls. Fun way to take advantage of `async` support in Jinja.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681375466, https://github.com/simonw/datasette/issues/943#issuecomment-675787416,https://api.github.com/repos/simonw/datasette/issues/943,675787416,MDEyOklzc3VlQ29tbWVudDY3NTc4NzQxNg==,9599,2020-08-19T00:42:38Z,2020-08-19T00:42:38Z,OWNER,"I just realised that this mechanism is kind of like being able to use microservices - make API calls within your application - except that everything runs in the same process against SQLite databases so calls will be _lightning fast_. It also means that a plugin can add a new internal API to Datasette that's accessible to other plugins by registering a new route with `register_routes`!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681375466, https://github.com/simonw/datasette/issues/943#issuecomment-675753114,https://api.github.com/repos/simonw/datasette/issues/943,675753114,MDEyOklzc3VlQ29tbWVudDY3NTc1MzExNA==,9599,2020-08-18T22:34:55Z,2020-08-18T22:34:55Z,OWNER,"Maybe allow this: response = await datasette.get(""/{database}/{table}.json"", database=database, table=table) This could cause problems if users ever need to pass literal `{` in their paths. Maybe allow this too: response = await datasette.get(""/{database}/{table}.json"", interpolate=False) Not convinced this is useful - it's a bit unintuitive.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681375466, https://github.com/simonw/datasette/issues/943#issuecomment-675752436,https://api.github.com/repos/simonw/datasette/issues/943,675752436,MDEyOklzc3VlQ29tbWVudDY3NTc1MjQzNg==,9599,2020-08-18T22:32:44Z,2020-08-18T22:32:44Z,OWNER,"One thing to consider here: Datasette's table and database name escaping rules can be a little bit convoluted. If a plugin wants to get back the first five rows of a table, it will need to construct a URL `/dbname/tablename?_size=5` - but it will need to know how to turn the database and table names into the correctly escaped `dbname` and `tablename` values. Here's how the `row.html` table handles that right now: https://github.com/simonw/datasette/blob/b21ed237ab940768574c834aa5a7130724bd3a2d/datasette/templates/row.html#L19-L23 It would be an improvement to have this logic abstracted out somewhere and documented so plugins can use it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681375466, https://github.com/simonw/datasette/issues/943#issuecomment-675751719,https://api.github.com/repos/simonw/datasette/issues/943,675751719,MDEyOklzc3VlQ29tbWVudDY3NTc1MTcxOQ==,9599,2020-08-18T22:30:27Z,2020-08-18T22:30:27Z,OWNER,"Right now calling `datasette.app()` instantiates an ASGI application - complete with a bunch of routes and wrappers - and returns that application object. Calling it twice instantiates another ASGI application. I think a single `Datasette` instance should only ever create a single ASGI app - so the `.app()` method should cache the ASGI app that it returns the first time and return the same application again on future calls.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681375466, https://github.com/simonw/datasette/issues/915#issuecomment-675751136,https://api.github.com/repos/simonw/datasette/issues/915,675751136,MDEyOklzc3VlQ29tbWVudDY3NTc1MTEzNg==,9599,2020-08-18T22:28:36Z,2020-08-18T22:28:36Z,OWNER,I'm closing this in favour of an internal requests mechanism in #943.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",671763164,