html_url,issue_url,id,node_id,user,user_label,created_at,updated_at,author_association,body,reactions,issue,issue_label,performed_via_github_app https://github.com/simonw/sqlite-utils/issues/196#issuecomment-722070569,https://api.github.com/repos/simonw/sqlite-utils/issues/196,722070569,MDEyOklzc3VlQ29tbWVudDcyMjA3MDU2OQ==,9599,simonw,2020-11-05T01:38:40Z,2020-11-05T01:38:40Z,OWNER,"I'm going to try `re.VERBOSE` to see if I can make this readable with comments. https://docs.python.org/3/howto/regex.html ```python charref = re.compile(r"""""" &[#] # Start of a numeric entity reference ( 0[0-7]+ # Octal form | [0-9]+ # Decimal form | x[0-9a-fA-F]+ # Hexadecimal form ) ; # Trailing semicolon """""", re.VERBOSE) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",736520310,Introspect if table is FTS4 or FTS5, https://github.com/simonw/sqlite-utils/issues/196#issuecomment-722064258,https://api.github.com/repos/simonw/sqlite-utils/issues/196,722064258,MDEyOklzc3VlQ29tbWVudDcyMjA2NDI1OA==,9599,simonw,2020-11-05T01:18:07Z,2020-11-05T01:21:31Z,OWNER,"``` In [8]: r = re.compile(r""""""'[^']*(?:''[^']*)*'"""""") In [9]: r.match(""'fo'o'"") Out[9]: In [10]: r.match(""'fo''o'"") Out[10]: ``` `'[^']*(?:''[^']*)*'` This matches a single quote, then 0+ not-single-quotes, then 0+ (either 0+ not-single quotes or a double single quote), then a single quote. Unrolling the loop technique described here: http://www.softec.lu/site/RegularExpressions/UnrollingTheLoop","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",736520310,Introspect if table is FTS4 or FTS5, https://github.com/simonw/sqlite-utils/issues/196#issuecomment-722062449,https://api.github.com/repos/simonw/sqlite-utils/issues/196,722062449,MDEyOklzc3VlQ29tbWVudDcyMjA2MjQ0OQ==,9599,simonw,2020-11-05T01:12:14Z,2020-11-05T01:12:14Z,OWNER,"Good news: I don't think I have to deal with `foo.tablename`, because that doesn't get reflected in the `sqlite_master` table: ``` sqlite> attach 'foo.db' as foo; sqlite> create table foo.`bant` (id int); sqlite> select * from foo.sqlite_master; table|bant|bant|2|CREATE TABLE `bant` (id int) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",736520310,Introspect if table is FTS4 or FTS5, https://github.com/simonw/sqlite-utils/issues/196#issuecomment-722062082,https://api.github.com/repos/simonw/sqlite-utils/issues/196,722062082,MDEyOklzc3VlQ29tbWVudDcyMjA2MjA4Mg==,9599,simonw,2020-11-05T01:10:51Z,2020-11-05T01:10:51Z,OWNER,"I confirmed all three of these are valid syntax for creating tables: ``` ~ % sqlite3 tmp.db SQLite version 3.28.0 2019-04-15 14:49:49 Enter "".help"" for usage hints. sqlite> create table 'foo''and' (id int); sqlite> create table ""bar""""and"" (id int); sqlite> create table [baz] (id int); sqlite> create table `bant` (id int); sqlite> .schema CREATE TABLE IF NOT EXISTS 'foo''and' (id int); CREATE TABLE IF NOT EXISTS ""bar""""and"" (id int); CREATE TABLE [baz] (id int); CREATE TABLE `bant` (id int); sqlite> select * from sqlite_master; table|foo'and|foo'and|2|CREATE TABLE 'foo''and' (id int) table|bar""and|bar""and|3|CREATE TABLE ""bar""""and"" (id int) table|baz|baz|4|CREATE TABLE [baz] (id int) table|bant|bant|5|CREATE TABLE `bant` (id int) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",736520310,Introspect if table is FTS4 or FTS5, https://github.com/simonw/sqlite-utils/issues/196#issuecomment-722058598,https://api.github.com/repos/simonw/sqlite-utils/issues/196,722058598,MDEyOklzc3VlQ29tbWVudDcyMjA1ODU5OA==,9599,simonw,2020-11-05T00:59:58Z,2020-11-05T00:59:58Z,OWNER,"That two-in-a-row thing works for `""` too: https://latest.datasette.io/fixtures?sql=select+%22foo%22%2C+%27bar%27%2C+%22foo%22%22and%22%2C+%27bar%27%27and%27 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",736520310,Introspect if table is FTS4 or FTS5, https://github.com/simonw/sqlite-utils/issues/196#issuecomment-722057923,https://api.github.com/repos/simonw/sqlite-utils/issues/196,722057923,MDEyOklzc3VlQ29tbWVudDcyMjA1NzkyMw==,9599,simonw,2020-11-05T00:57:22Z,2020-11-05T00:57:22Z,OWNER,"Then https://sqlite.org/lang_expr.html#literal_values_constants_ says: > A string constant is formed by enclosing the string in single quotes ('). A single quote within the string can be encoded by putting two single quotes in a row - as in Pascal. C-style escapes using the backslash character are not supported because they are not standard SQL. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",736520310,Introspect if table is FTS4 or FTS5, https://github.com/simonw/sqlite-utils/issues/196#issuecomment-722057392,https://api.github.com/repos/simonw/sqlite-utils/issues/196,722057392,MDEyOklzc3VlQ29tbWVudDcyMjA1NzM5Mg==,9599,simonw,2020-11-05T00:55:31Z,2020-11-05T00:55:51Z,OWNER,"https://sqlite.org/lang_keywords.html says: > There are four ways of quoting keywords in SQLite: > > **'keyword'** A keyword in single quotes is a string literal. > **""keyword""** A keyword in double-quotes is an identifier. > **[keyword]** A keyword enclosed in square brackets is an identifier. This is not standard SQL. This quoting mechanism is used by MS Access and SQL Server and is included in SQLite for compatibility. > **\`keyword\`** A keyword enclosed in grave accents (ASCII code 96) is an identifier. This is not standard SQL. This quoting mechanism is used by MySQL and is included in SQLite for compatibility.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",736520310,Introspect if table is FTS4 or FTS5, https://github.com/simonw/sqlite-utils/issues/196#issuecomment-722056576,https://api.github.com/repos/simonw/sqlite-utils/issues/196,722056576,MDEyOklzc3VlQ29tbWVudDcyMjA1NjU3Ng==,9599,simonw,2020-11-05T00:52:42Z,2020-11-05T00:52:42Z,OWNER,"I could use a parsing library like https://parsy.readthedocs.io/en/latest/tutorial.html for this - or `pyparsing` which has a SQLite example here: https://github.com/pyparsing/pyparsing/blob/master/examples/select_parser.py I'd rather not add a new dependency for this though so I'm going to see if I can get something that's good-enough just using a regular expression.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",736520310,Introspect if table is FTS4 or FTS5, https://github.com/simonw/sqlite-utils/issues/196#issuecomment-722055291,https://api.github.com/repos/simonw/sqlite-utils/issues/196,722055291,MDEyOklzc3VlQ29tbWVudDcyMjA1NTI5MQ==,9599,simonw,2020-11-05T00:48:10Z,2020-11-05T00:48:10Z,OWNER,This is blocking landing `.search()` in #195,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",736520310,Introspect if table is FTS4 or FTS5, https://github.com/simonw/sqlite-utils/issues/196#issuecomment-722055104,https://api.github.com/repos/simonw/sqlite-utils/issues/196,722055104,MDEyOklzc3VlQ29tbWVudDcyMjA1NTEwNA==,9599,simonw,2020-11-05T00:47:34Z,2020-11-05T00:47:34Z,OWNER,"This is surprisingly difficult. I need to parse the `CREATE VIRTUAL TABLE` statement, which will look something like this: ```sql CREATE VIRTUAL TABLE ""global-power-plants_fts"" USING FTS5 (""name"", content=""global-power-plants"") ``` The problem is I need to be able to handle various different quoting formats for the table name (`mytable` v.s. `""mytable""` v.s. `[mytable]`) plus I need to look out for `CREATE TABLE IF NOT EXISTS`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",736520310,Introspect if table is FTS4 or FTS5, https://github.com/simonw/sqlite-utils/issues/192#issuecomment-722054264,https://api.github.com/repos/simonw/sqlite-utils/issues/192,722054264,MDEyOklzc3VlQ29tbWVudDcyMjA1NDI2NA==,9599,simonw,2020-11-05T00:44:39Z,2020-11-05T00:44:39Z,OWNER,"I want `.search()` to work against both FTS5 and FTS4 tables - but sort by rank should only work for FTS5. This means I need to be able to introspect and tell if a table is FTS4 or FTS5.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",735532751,sqlite-utils search command, https://github.com/simonw/datasette/issues/1082#issuecomment-721931504,https://api.github.com/repos/simonw/datasette/issues/1082,721931504,MDEyOklzc3VlQ29tbWVudDcyMTkzMTUwNA==,9599,simonw,2020-11-04T19:32:47Z,2020-11-04T19:35:44Z,OWNER,"I wonder if setting a soft memory limit within Datasette would help here: https://www.sqlite.org/malloc.html#_setting_memory_usage_limits > If attempts are made to allocate more memory than specified by the soft heap limit, then SQLite will first attempt to free cache memory before continuing with the allocation request. https://www.sqlite.org/pragma.html#pragma_soft_heap_limit > **PRAGMA soft_heap_limit** > **PRAGMA soft_heap_limit=N** > > This pragma invokes the [sqlite3_soft_heap_limit64()](https://www.sqlite.org/c3ref/hard_heap_limit64.html) interface with the argument N, if N is specified and is a non-negative integer. The soft_heap_limit pragma always returns the same integer that would be returned by the [sqlite3_soft_heap_limit64](https://www.sqlite.org/c3ref/hard_heap_limit64.html)(-1) C-language function.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",735852274,DigitalOcean buildpack memory errors for large sqlite db?, https://github.com/simonw/datasette/issues/1083#issuecomment-721927254,https://api.github.com/repos/simonw/datasette/issues/1083,721927254,MDEyOklzc3VlQ29tbWVudDcyMTkyNzI1NA==,9599,simonw,2020-11-04T19:24:34Z,2020-11-04T19:24:34Z,OWNER,"Related: #856 - if it's possible to paginate correctly configured canned query then the CSV option to ""stream all rows"" could work for queries as well as tables.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",736365306,Advanced CSV export for arbitrary queries, https://github.com/simonw/datasette/issues/1083#issuecomment-721926827,https://api.github.com/repos/simonw/datasette/issues/1083,721926827,MDEyOklzc3VlQ29tbWVudDcyMTkyNjgyNw==,9599,simonw,2020-11-04T19:23:42Z,2020-11-04T19:23:42Z,OWNER,"https://latest.datasette.io/fixtures/sortable#export has advanced export options, but https://latest.datasette.io/fixtures?sql=select+pk1%2C+pk2%2C+content%2C+sortable%2C+sortable_with_nulls%2C+sortable_with_nulls_2%2C+text+from+sortable+order+by+pk1%2C+pk2+limit+101 does not.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",736365306,Advanced CSV export for arbitrary queries, https://github.com/simonw/datasette/issues/268#issuecomment-721896822,https://api.github.com/repos/simonw/datasette/issues/268,721896822,MDEyOklzc3VlQ29tbWVudDcyMTg5NjgyMg==,9599,simonw,2020-11-04T18:23:29Z,2020-11-04T18:23:29Z,OWNER,"Worth noting that joining to get the rank works for FTS5 but not for FTS4 - see comment here: https://github.com/simonw/sqlite-utils/issues/192#issuecomment-721420539 Easiest solution would be to only support sort-by-rank for FTS5 tables. Alternative would be to depend on https://github.com/simonw/sqlite-fts4","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",323718842,Mechanism for ranking results from SQLite full-text search, https://github.com/simonw/datasette/issues/1082#issuecomment-721547177,https://api.github.com/repos/simonw/datasette/issues/1082,721547177,MDEyOklzc3VlQ29tbWVudDcyMTU0NzE3Nw==,39538958,justmars,2020-11-04T06:52:30Z,2020-11-04T06:53:16Z,NONE,"I think I tried the same db size on the following scenarios in Digital Ocean: 1. Basic ($5/month) with 512MB RAM 2. Basic ($10/month) with 1GB RAM 3. Pro ($12/month) with 1GB RAM All such attempts conked out with ""out of memory"" errors","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",735852274,DigitalOcean buildpack memory errors for large sqlite db?, https://github.com/simonw/datasette/issues/1082#issuecomment-721545090,https://api.github.com/repos/simonw/datasette/issues/1082,721545090,MDEyOklzc3VlQ29tbWVudDcyMTU0NTA5MA==,9599,simonw,2020-11-04T06:47:15Z,2020-11-04T06:47:15Z,OWNER,"I've run into a similar problem with Google Cloud Run: beyond a certain size of database file I find myself needing to run instances there with more RAM assigned to them. I haven't yet figured out a method to estimate the amount of RAM that will be needed to successfully serve a database file of a specific size- I've been using trial and error. 5GB is quite a big database file, so it doesn't surprise me that it may need a bigger instance. I recommend trying it on a 1GB or 2GB of RAM Digital Ocean instance (their default is 512MB) and see if that works. Let me know what you find out!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",735852274,DigitalOcean buildpack memory errors for large sqlite db?, https://github.com/simonw/sqlite-utils/issues/192#issuecomment-721453779,https://api.github.com/repos/simonw/sqlite-utils/issues/192,721453779,MDEyOklzc3VlQ29tbWVudDcyMTQ1Mzc3OQ==,9599,simonw,2020-11-04T00:59:24Z,2020-11-04T00:59:36Z,OWNER,"FTS5 was added in SQLite 3.9.0 in 2015-10-14 - so about a year after CTEs, which means CTEs will always be safe to use with FTS5 queries.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",735532751,sqlite-utils search command, https://github.com/simonw/sqlite-utils/issues/192#issuecomment-721420907,https://api.github.com/repos/simonw/sqlite-utils/issues/192,721420907,MDEyOklzc3VlQ29tbWVudDcyMTQyMDkwNw==,9599,simonw,2020-11-03T23:07:01Z,2020-11-03T23:07:01Z,OWNER,"I could depend on my `sqlite-fts4` library to solve this: https://github.com/simonw/sqlite-fts4 Or I could say that only `FTS5` is supported for ranked searches - but still support `.search()` against FTS4 just without the option to sort by relevance.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",735532751,sqlite-utils search command, https://github.com/simonw/sqlite-utils/issues/192#issuecomment-721420539,https://api.github.com/repos/simonw/sqlite-utils/issues/192,721420539,MDEyOklzc3VlQ29tbWVudDcyMTQyMDUzOQ==,9599,simonw,2020-11-03T23:05:53Z,2020-11-03T23:05:53Z,OWNER,"Just realized there's a problem with the SQL I am using here: joining to get `rank` in this way only works against FTS5 tables, it doesn't work against FTS4. https://github.com/simonw/sqlite-utils/blob/c8a900df59efd34f394c863c0adff9912f1bf1d7/sqlite_utils/db.py#L1303-L1319","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",735532751,sqlite-utils search command, https://github.com/simonw/sqlite-utils/pull/195#issuecomment-721397665,https://api.github.com/repos/simonw/sqlite-utils/issues/195,721397665,MDEyOklzc3VlQ29tbWVudDcyMTM5NzY2NQ==,9599,simonw,2020-11-03T22:02:57Z,2020-11-03T22:02:57Z,OWNER,Documentation so far: https://github.com/simonw/sqlite-utils/blob/6cadc6103ff1ba58c6409ce7fba74259e72965d9/docs/cli.rst#executing-searches,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",735663855,table.search() improvements plus sqlite-utils search command, https://github.com/simonw/sqlite-utils/issues/192#issuecomment-721319602,https://api.github.com/repos/simonw/sqlite-utils/issues/192,721319602,MDEyOklzc3VlQ29tbWVudDcyMTMxOTYwMg==,9599,simonw,2020-11-03T19:05:05Z,2020-11-03T19:05:05Z,OWNER,"Relevant example using a SQLite CTE: https://github.com/simonw/datasette/issues/268#issuecomment-675725464 CTEs were added in SQLite 3.8.3 in 2014-02-03 so they should be safe to use. If someone tries to run `sqlite-utils search` no an older version of SQLite they'll get an error, which I think is OK.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",735532751,sqlite-utils search command, https://github.com/simonw/datasette/issues/596#issuecomment-720741903,https://api.github.com/repos/simonw/datasette/issues/596,720741903,MDEyOklzc3VlQ29tbWVudDcyMDc0MTkwMw==,132978,terrycojones,2020-11-02T21:44:45Z,2020-11-02T21:44:45Z,NONE,Hi & thanks for the note @simonw! I wish I had more time to play with (and contribute to) datasette. I know you don't need me to tell you that it's super cool :-),"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",507454958,Handle really wide tables better, https://github.com/simonw/datasette/issues/830#issuecomment-720700065,https://api.github.com/repos/simonw/datasette/issues/830,720700065,MDEyOklzc3VlQ29tbWVudDcyMDcwMDA2NQ==,9599,simonw,2020-11-02T20:15:36Z,2020-11-02T20:15:36Z,OWNER,"#427 had a bunch of ambitious plans for faceting that I haven't realized yet: > Think of all of the potential kinds of facets: > > * `?_facet_array=tags` where tags is a JSON array of values > * `_facet_date=datetimecol` - faceted by date part of a datetime > * `_facet_bins=numeric_column` - can I do some kind of fancy binning here? Might need to take an argument > * `?_facet_bins=numeric_column:5` - could be a way to take an argument. We’ll ignore columns with a : in their name. > * `?_facet_json=jsoncol:jsonpath` - could use a JSON path to extract out something to facet on? > * `?_facet_percentile=numericcolumn` - could this work? > * `?_facet_function=column:sqlfunctionname` - maybe this could be interesting? Would allow for e.g. facet by soundex > * `?_facet_prefix=column:prefix` - facet by terms but only if they start with a specific prefix > * `?_facet_substring=column:3,6` - facet by a substr(column, 3, 6)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",636511683,Redesign register_facet_classes plugin hook, https://github.com/simonw/datasette/issues/1080#issuecomment-720696827,https://api.github.com/repos/simonw/datasette/issues/1080,720696827,MDEyOklzc3VlQ29tbWVudDcyMDY5NjgyNw==,9599,simonw,2020-11-02T20:08:49Z,2020-11-02T20:13:56Z,OWNER,"Implementing pagination for facets will be interesting. Would be easier if I had a nicer reusable internal pagination mechanism, which is also needed for #856 (pagination of canned queries).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",734777631,"""View all"" option for facets, to provide a (paginated) list of ALL of the facet counts plus a link to view them", https://github.com/simonw/datasette/issues/1080#issuecomment-720699160,https://api.github.com/repos/simonw/datasette/issues/1080,720699160,MDEyOklzc3VlQ29tbWVudDcyMDY5OTE2MA==,9599,simonw,2020-11-02T20:13:42Z,2020-11-02T20:13:42Z,OWNER,Also relevant to this issue: #830 - redesigning the facet plugin hook in preparation for Datasette 1.0. And #972 supporting faceting against arbitrary queries.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",734777631,"""View all"" option for facets, to provide a (paginated) list of ALL of the facet counts plus a link to view them", https://github.com/simonw/datasette/issues/1080#issuecomment-720698577,https://api.github.com/repos/simonw/datasette/issues/1080,720698577,MDEyOklzc3VlQ29tbWVudDcyMDY5ODU3Nw==,9599,simonw,2020-11-02T20:12:26Z,2020-11-02T20:12:26Z,OWNER,"For regular column faceting, here's the query that is used: https://github.com/simonw/datasette/blob/13d1228d80c91d382a05b1a9549ed02c300ef851/datasette/facets.py#L196-L204 Since it uses `order by count desc, value` maybe those values could be used to implement cursor-based pagination. That wouldn't be robust in the face of changing data, but I'm not sure it's possible to implement paginated faceting in a way that survives ongoing changes to the underlying data.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",734777631,"""View all"" option for facets, to provide a (paginated) list of ALL of the facet counts plus a link to view them", https://github.com/simonw/datasette/issues/1080#issuecomment-720697226,https://api.github.com/repos/simonw/datasette/issues/1080,720697226,MDEyOklzc3VlQ29tbWVudDcyMDY5NzIyNg==,9599,simonw,2020-11-02T20:09:38Z,2020-11-02T20:09:38Z,OWNER,"Maybe this ends up being code that defers to a simulated canned query, rendered using the existing `query.html` template.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",734777631,"""View all"" option for facets, to provide a (paginated) list of ALL of the facet counts plus a link to view them", https://github.com/simonw/datasette/issues/1080#issuecomment-720695174,https://api.github.com/repos/simonw/datasette/issues/1080,720695174,MDEyOklzc3VlQ29tbWVudDcyMDY5NTE3NA==,9599,simonw,2020-11-02T20:05:26Z,2020-11-02T20:05:26Z,OWNER,"URL design: `/database/table/-/facet/colname` And for other types of facet (to be supported later): `/database/table/-/facet/colname?_type=m2m`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",734777631,"""View all"" option for facets, to provide a (paginated) list of ALL of the facet counts plus a link to view them", https://github.com/simonw/datasette/issues/596#issuecomment-720689653,https://api.github.com/repos/simonw/datasette/issues/596,720689653,MDEyOklzc3VlQ29tbWVudDcyMDY4OTY1Mw==,9599,simonw,2020-11-02T19:53:36Z,2020-11-02T19:53:47Z,OWNER,"In #998 I implemented a horizontal scrollbar for these tables, which is a big improvement - demo here: https://global-power-plants.datasettes.com/global-power-plants/global-power-plants","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",507454958,Handle really wide tables better, https://github.com/simonw/datasette/issues/1077#issuecomment-720654925,https://api.github.com/repos/simonw/datasette/issues/1077,720654925,MDEyOklzc3VlQ29tbWVudDcyMDY1NDkyNQ==,9599,simonw,2020-11-02T18:43:25Z,2020-11-02T18:43:25Z,OWNER,Demo: https://latest.datasette.io/fixtures?_bot=1,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733829385,database_actions plugin hook, https://github.com/simonw/datasette/issues/1077#issuecomment-720637322,https://api.github.com/repos/simonw/datasette/issues/1077,720637322,MDEyOklzc3VlQ29tbWVudDcyMDYzNzMyMg==,9599,simonw,2020-11-02T18:09:17Z,2020-11-02T18:09:17Z,OWNER,Here's the `table_actions` implementation: 2f7731e9e5ff9b324beb5039fbe2be55d704a184,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733829385,database_actions plugin hook, https://github.com/simonw/datasette/issues/838#issuecomment-720354227,https://api.github.com/repos/simonw/datasette/issues/838,720354227,MDEyOklzc3VlQ29tbWVudDcyMDM1NDIyNw==,82988,psychemedia,2020-11-02T09:33:58Z,2020-11-02T09:33:58Z,CONTRIBUTOR,"Thanks; just a note that the `datasette.urls.static(path)` and `datasette.urls.static_plugins(plugin_name, path)` items both seem to be repeated and appear in the docs twice?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",637395097,Incorrect URLs when served behind a proxy with base_url set, https://github.com/simonw/datasette/issues/1079#issuecomment-720110298,https://api.github.com/repos/simonw/datasette/issues/1079,720110298,MDEyOklzc3VlQ29tbWVudDcyMDExMDI5OA==,9599,simonw,2020-11-01T15:58:22Z,2020-11-01T15:58:22Z,OWNER,Might try a drop shadow on that menu too.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733999615,Handle long breadcrumbs better with new menu, https://github.com/simonw/datasette/issues/782#issuecomment-720028476,https://api.github.com/repos/simonw/datasette/issues/782,720028476,MDEyOklzc3VlQ29tbWVudDcyMDAyODQ3Ng==,9599,simonw,2020-11-01T05:00:05Z,2020-11-01T05:00:05Z,OWNER,This should be the key focus for Datasette 0.52.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format, https://github.com/simonw/datasette/issues/949#issuecomment-720021029,https://api.github.com/repos/simonw/datasette/issues/949,720021029,MDEyOklzc3VlQ29tbWVudDcyMDAyMTAyOQ==,9599,simonw,2020-11-01T03:29:48Z,2020-11-01T03:29:48Z,OWNER,I'm not going to do any more work on this - SQL isn't an auto-complete friendly enough language.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",684961449,Try out CodeMirror SQL hints, https://github.com/simonw/datasette/issues/1077#issuecomment-720003026,https://api.github.com/repos/simonw/datasette/issues/1077,720003026,MDEyOklzc3VlQ29tbWVudDcyMDAwMzAyNg==,9599,simonw,2020-10-31T23:48:21Z,2020-10-31T23:50:07Z,OWNER,Needed by https://github.com/simonw/datasette-backup/issues/6,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733829385,database_actions plugin hook, https://github.com/simonw/datasette/issues/1047#issuecomment-719994676,https://api.github.com/repos/simonw/datasette/issues/1047,719994676,MDEyOklzc3VlQ29tbWVudDcxOTk5NDY3Ng==,9599,simonw,2020-10-31T22:11:25Z,2020-10-31T22:11:25Z,OWNER,https://docs.datasette.io/en/latest/binary_data.html,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",728895233,A new section in the docs about how Datasette handles BLOB columns, https://github.com/simonw/datasette/issues/1027#issuecomment-719988113,https://api.github.com/repos/simonw/datasette/issues/1027,719988113,MDEyOklzc3VlQ29tbWVudDcxOTk4ODExMw==,9599,simonw,2020-10-31T21:03:37Z,2020-10-31T21:03:37Z,OWNER,"On my Mac, I run: datasette . -p 8009 --config base_url:/datasette-prefix/ Then I edited `/usr/local/etc/httpd/httpd.conf` and add this section: ``` LoadModule proxy_module lib/httpd/modules/mod_proxy.so LoadModule proxy_http_module lib/httpd/modules/mod_proxy_http.so ProxyPass /datasette-prefix/ http://localhost:8009/datasette-prefix/ ``` I ran Apache in the foreground like so: apachectl -X Now hitting http://localhost:8081/datasette-prefix/fixtures/compound_three_primary_keys worked!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722758132,Add documentation on serving Datasette behind a proxy using base_url, https://github.com/simonw/datasette/issues/1023#issuecomment-719986922,https://api.github.com/repos/simonw/datasette/issues/1023,719986922,MDEyOklzc3VlQ29tbWVudDcxOTk4NjkyMg==,9599,simonw,2020-10-31T20:51:01Z,2020-10-31T20:51:01Z,OWNER,This should all be working correctly now.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722673818,Fix issues relating to base_url, https://github.com/simonw/datasette/issues/838#issuecomment-719986904,https://api.github.com/repos/simonw/datasette/issues/838,719986904,MDEyOklzc3VlQ29tbWVudDcxOTk4NjkwNA==,9599,simonw,2020-10-31T20:50:41Z,2020-10-31T20:50:41Z,OWNER,"OK, this should be working now. You can use the `datasette.urls.static_plugins()` method to generate the correct URLs in the `extra_css_urls` plugin hook: https://docs.datasette.io/en/latest/internals.html#datasette-urls","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",637395097,Incorrect URLs when served behind a proxy with base_url set, https://github.com/simonw/datasette/issues/1041#issuecomment-719986800,https://api.github.com/repos/simonw/datasette/issues/1041,719986800,MDEyOklzc3VlQ29tbWVudDcxOTk4NjgwMA==,9599,simonw,2020-10-31T20:49:28Z,2020-10-31T20:49:28Z,OWNER,Implemented in a4ca26a2659d21779adf625183061d8879954c15,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",727627923,extra_js_urls and extra_css_urls should respect base_url setting, https://github.com/simonw/datasette/issues/1072#issuecomment-719986698,https://api.github.com/repos/simonw/datasette/issues/1072,719986698,MDEyOklzc3VlQ29tbWVudDcxOTk4NjY5OA==,9599,simonw,2020-10-31T20:48:17Z,2020-10-31T20:48:17Z,OWNER,Here's the `datasette-edit-templates` plugin WIP I had before removing the hook: https://github.com/simonw/datasette-edit-templates/tree/82855c2612b84bc09c48fca885f831633a0d1552,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733499930,load_template hook doesn't work for include/extends, https://github.com/simonw/datasette/issues/1075#issuecomment-719983750,https://api.github.com/repos/simonw/datasette/issues/1075,719983750,MDEyOklzc3VlQ29tbWVudDcxOTk4Mzc1MA==,9599,simonw,2020-10-31T20:22:29Z,2020-10-31T20:22:29Z,OWNER,I bet this is because I'm mucking around with one of those `__` methods. I'll try just doing the non-underscore methods instead.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733796942,PrefixedUrlString mechanism broke everything, https://github.com/simonw/datasette/issues/1075#issuecomment-719983565,https://api.github.com/repos/simonw/datasette/issues/1075,719983565,MDEyOklzc3VlQ29tbWVudDcxOTk4MzU2NQ==,9599,simonw,2020-10-31T20:21:03Z,2020-10-31T20:21:03Z,OWNER,"Here's the output of `dir(str)`: `['__add__', '__class__', '__contains__', '__delattr__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__getitem__', '__getnewargs__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__iter__', '__le__', '__len__', '__lt__', '__mod__', '__mul__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__rmod__', '__rmul__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', 'capitalize', 'casefold', 'center', 'count', 'encode', 'endswith', 'expandtabs', 'find', 'format', 'format_map', 'index', 'isalnum', 'isalpha', 'isascii', 'isdecimal', 'isdigit', 'isidentifier', 'islower', 'isnumeric', 'isprintable', 'isspace', 'istitle', 'isupper', 'join', 'ljust', 'lower', 'lstrip', 'maketrans', 'partition', 'replace', 'rfind', 'rindex', 'rjust', 'rpartition', 'rsplit', 'rstrip', 'split', 'splitlines', 'startswith', 'strip', 'swapcase', 'title', 'translate', 'upper', 'zfill']`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733796942,PrefixedUrlString mechanism broke everything, https://github.com/simonw/datasette/issues/1075#issuecomment-719983484,https://api.github.com/repos/simonw/datasette/issues/1075,719983484,MDEyOklzc3VlQ29tbWVudDcxOTk4MzQ4NA==,9599,simonw,2020-10-31T20:20:28Z,2020-10-31T20:20:28Z,OWNER,"It looks like this is specific to the way `PrefixedUrlString` is built. ``` (Pdb) class Weird(str): pass (Pdb) isinstance(Weird('bob'), collections.abc.Awaitable) False ``` So subclassing strings doesn't trigger this bug, but something about `PrefixedUrlString` causes the problem. Here's the current `PrefixedUrlString` implementation: https://github.com/simonw/datasette/blob/bf18b9ba175a7b25fb8b765847397dd6efb8bb7b/datasette/utils/__init__.py#L1015-L1035","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733796942,PrefixedUrlString mechanism broke everything, https://github.com/simonw/datasette/issues/1075#issuecomment-719983240,https://api.github.com/repos/simonw/datasette/issues/1075,719983240,MDEyOklzc3VlQ29tbWVudDcxOTk4MzI0MA==,9599,simonw,2020-10-31T20:18:49Z,2020-10-31T20:18:49Z,OWNER,"Here's the core problem: ``` (Pdb) isinstance('bob', collections.abc.Awaitable) False (Pdb) isinstance(PrefixedUrlString('bob'), collections.abc.Awaitable) *** TypeError: issubclass() arg 1 must be a class ``` For some reason `isinstance()` does not like being handed an instance of PrefixedUrlString.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733796942,PrefixedUrlString mechanism broke everything, https://github.com/simonw/datasette/issues/1075#issuecomment-719981173,https://api.github.com/repos/simonw/datasette/issues/1075,719981173,MDEyOklzc3VlQ29tbWVudDcxOTk4MTE3Mw==,9599,simonw,2020-10-31T20:02:30Z,2020-10-31T20:03:45Z,OWNER,"I wonder how Jinja's `Markup()` class works? It uses https://pypi.org/project/MarkupSafe/ It's a subclass of `str`, defined here: https://github.com/pallets/markupsafe/blob/master/src/markupsafe/__init__.py","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733796942,PrefixedUrlString mechanism broke everything, https://github.com/simonw/datasette/issues/1075#issuecomment-719980742,https://api.github.com/repos/simonw/datasette/issues/1075,719980742,MDEyOklzc3VlQ29tbWVudDcxOTk4MDc0Mg==,9599,simonw,2020-10-31T19:58:57Z,2020-10-31T19:58:57Z,OWNER,"Sample traceback: ``` /opt/hostedtoolcache/Python/3.7.9/x64/lib/python3.7/site-packages/jinja2/asyncsupport.py:173: in auto_await if inspect.isawaitable(value): /opt/hostedtoolcache/Python/3.7.9/x64/lib/python3.7/inspect.py:226: in isawaitable isinstance(object, collections.abc.Awaitable)) /opt/hostedtoolcache/Python/3.7.9/x64/lib/python3.7/abc.py:139: in __instancecheck__ return _abc_instancecheck(cls, instance) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ cls = subclass = .method of '/-/static/app.css'> def __subclasscheck__(cls, subclass): """"""Override for issubclass(subclass, cls)."""""" > return _abc_subclasscheck(cls, subclass) E TypeError: issubclass() arg 1 must be a class ``` This is within Jinja. It looks like Jinja really doesn't like methods that return non-string objects like `PrefixedUrlString`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733796942,PrefixedUrlString mechanism broke everything, https://github.com/simonw/datasette/issues/1074#issuecomment-719977864,https://api.github.com/repos/simonw/datasette/issues/1074,719977864,MDEyOklzc3VlQ29tbWVudDcxOTk3Nzg2NA==,9599,simonw,2020-10-31T19:35:01Z,2020-10-31T19:35:01Z,OWNER,"These plugins were not designed to be actually hosted online, so they do some nasty things like linking to the made-up `plugin-example.com` domain. I should fix that. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733768037,latest.datasette.io should include plugins from fixtures, https://github.com/simonw/datasette/issues/1067#issuecomment-719966176,https://api.github.com/repos/simonw/datasette/issues/1067,719966176,MDEyOklzc3VlQ29tbWVudDcxOTk2NjE3Ng==,9599,simonw,2020-10-31T17:51:31Z,2020-10-31T17:51:31Z,OWNER,"Demo: - https://latest.datasette.io/fixtures/facetable?_bot=1 - https://latest.datasette.io/fixtures/simple_view?_bot=1","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732905360,"Table actions menu on view pages, not on query pages", https://github.com/simonw/datasette/issues/1074#issuecomment-719965426,https://api.github.com/repos/simonw/datasette/issues/1074,719965426,MDEyOklzc3VlQ29tbWVudDcxOTk2NTQyNg==,9599,simonw,2020-10-31T17:45:00Z,2020-10-31T17:45:00Z,OWNER,"This is working. Go to https://latest.datasette.io/login-as-root and click the button, then visit this page to see extra content added by plugins: https://latest.datasette.io/fixtures/compound_three_primary_keys ![latest-plugins](https://user-images.githubusercontent.com/9599/97786052-19e53d00-1b66-11eb-8268-b452e08965e3.gif) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733768037,latest.datasette.io should include plugins from fixtures, https://github.com/simonw/datasette/issues/1074#issuecomment-719963074,https://api.github.com/repos/simonw/datasette/issues/1074,719963074,MDEyOklzc3VlQ29tbWVudDcxOTk2MzA3NA==,9599,simonw,2020-10-31T17:23:48Z,2020-10-31T17:23:48Z,OWNER,"Needs a way to login as root, seeing as several plugins only show extra content if the user is logged in as root.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733768037,latest.datasette.io should include plugins from fixtures, https://github.com/simonw/datasette/issues/1067#issuecomment-719961701,https://api.github.com/repos/simonw/datasette/issues/1067,719961701,MDEyOklzc3VlQ29tbWVudDcxOTk2MTcwMQ==,9599,simonw,2020-10-31T17:11:59Z,2020-10-31T17:11:59Z,OWNER,It bothers me that these aren't visible in any public demos. Maybe `latest.datasette.io` should include the `my_plugins.py` and `my_plugins2.py` plugins?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732905360,"Table actions menu on view pages, not on query pages", https://github.com/simonw/datasette/issues/1026#issuecomment-719959754,https://api.github.com/repos/simonw/datasette/issues/1026,719959754,MDEyOklzc3VlQ29tbWVudDcxOTk1OTc1NA==,9599,simonw,2020-10-31T16:56:35Z,2020-10-31T16:56:35Z,OWNER,#1041 can also benefit from the string subclass that shows that `base_url` has been added.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722738988,How should datasette.client interact with base_url, https://github.com/simonw/datasette/issues/1067#issuecomment-719959419,https://api.github.com/repos/simonw/datasette/issues/1067,719959419,MDEyOklzc3VlQ29tbWVudDcxOTk1OTQxOQ==,9599,simonw,2020-10-31T16:53:42Z,2020-10-31T16:53:42Z,OWNER,For the 0.51 release I'm going to add tests that show this works on view pages. I won't implement it for query pages.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732905360,"Table actions menu on view pages, not on query pages", https://github.com/simonw/datasette/issues/1067#issuecomment-719956184,https://api.github.com/repos/simonw/datasette/issues/1067,719956184,MDEyOklzc3VlQ29tbWVudDcxOTk1NjE4NA==,9599,simonw,2020-10-31T16:26:09Z,2020-10-31T16:26:09Z,OWNER,"Should the hook provide an indication that it's running on a different type of page? I think yes for queries. Not sure about views - they behave very much like tables, and the plugin can always introspect to see if something is a view if it needs to.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732905360,"Table actions menu on view pages, not on query pages", https://github.com/simonw/datasette/issues/1070#issuecomment-719955724,https://api.github.com/repos/simonw/datasette/issues/1070,719955724,MDEyOklzc3VlQ29tbWVudDcxOTk1NTcyNA==,9599,simonw,2020-10-31T16:22:45Z,2020-10-31T16:22:45Z,OWNER,I've removed this plugin hook in #1073.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733390884,load_template() example in documentation showing loading from a database, https://github.com/simonw/datasette/issues/1072#issuecomment-719955491,https://api.github.com/repos/simonw/datasette/issues/1072,719955491,MDEyOklzc3VlQ29tbWVudDcxOTk1NTQ5MQ==,9599,simonw,2020-10-31T16:20:58Z,2020-10-31T16:20:58Z,OWNER,"Here's the proof of concept `FunctionLoader` that showed me that this wasn't going to work: ```diff diff --git a/datasette/app.py b/datasette/app.py index 4b28e71..b076be7 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -21,7 +21,7 @@ from pathlib import Path from markupsafe import Markup from itsdangerous import URLSafeSerializer import jinja2 -from jinja2 import ChoiceLoader, Environment, FileSystemLoader, PrefixLoader +from jinja2 import ChoiceLoader, Environment, FileSystemLoader, FunctionLoader, PrefixLoader from jinja2.environment import Template from jinja2.exceptions import TemplateNotFound import uvicorn @@ -300,6 +300,7 @@ class Datasette: template_paths.append(default_templates) template_loader = ChoiceLoader( [ + FunctionLoader(self._load_template_from_plugins), FileSystemLoader(template_paths), # Support {% extends ""default:table.html"" %}: PrefixLoader( @@ -322,6 +323,17 @@ class Datasette: self._root_token = secrets.token_hex(32) self.client = DatasetteClient(self) + def _load_template_from_plugins(self, template): + # ""If auto reloading is enabled it’s called to check if the template changed"" + uptodatefunc = lambda: True + source = pm.hook.load_template( + template=template, + datasette=self, + ) + if source is None: + return None + return source, template, uptodatefunc + @property def urls(self): return Urls(self) @@ -719,35 +731,7 @@ class Datasette: else: if isinstance(templates, str): templates = [templates] - - # Give plugins first chance at loading the template - break_outer = False - plugin_template_source = None - plugin_template_name = None - template_name = None - for template_name in templates: - if break_outer: - break - plugin_template_source = pm.hook.load_template( - template=template_name, - request=request, - datasette=self, - ) - plugin_template_source = await await_me_maybe(plugin_template_source) - if plugin_template_source: - break_outer = True - plugin_template_name = template_name - break - if plugin_template_source is not None: - template = self.jinja_env.from_string(plugin_template_source) - else: - template = self.jinja_env.select_template(templates) - for template_name in templates: - from_plugin = template_name == plugin_template_name - used = from_plugin or template_name == template.name - templates_considered.append( - {""name"": template_name, ""used"": used, ""from_plugin"": from_plugin} - ) + template = self.jinja_env.select_template(templates) body_scripts = [] # pylint: disable=no-member for extra_script in pm.hook.extra_body_script( diff --git a/datasette/hookspecs.py b/datasette/hookspecs.py index ca84b35..7804def 100644 --- a/datasette/hookspecs.py +++ b/datasette/hookspecs.py @@ -50,7 +50,7 @@ def extra_template_vars( @hookspec(firstresult=True) -def load_template(template, request, datasette): +def load_template(template, datasette): ""Load the specified template, returning the template code as a string"" diff --git a/docs/plugin_hooks.rst b/docs/plugin_hooks.rst index 3c57b6a..8f2704e 100644 --- a/docs/plugin_hooks.rst +++ b/docs/plugin_hooks.rst @@ -273,15 +273,12 @@ Example: `datasette-cluster-map >> def load_template(name): ... if name == 'index.html': ... return '...' ... >>> loader = FunctionLoader(load_template) ``` Just one catch: I need to be able to load templates asynchronously, because they live in the database. Let's hope Jinja has a mechanism for that!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733499930,load_template hook doesn't work for include/extends, https://github.com/simonw/datasette/issues/1072#issuecomment-719785005,https://api.github.com/repos/simonw/datasette/issues/1072,719785005,MDEyOklzc3VlQ29tbWVudDcxOTc4NTAwNQ==,9599,simonw,2020-10-30T20:36:22Z,2020-10-30T20:36:22Z,OWNER,"It should be easy enough to show a comment that says which original template names were considered, but I may not be able to show which one was actually used (or which ones came from plugins).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733499930,load_template hook doesn't work for include/extends, https://github.com/simonw/datasette/issues/1072#issuecomment-719784606,https://api.github.com/repos/simonw/datasette/issues/1072,719784606,MDEyOklzc3VlQ29tbWVudDcxOTc4NDYwNg==,9599,simonw,2020-10-30T20:35:33Z,2020-10-30T20:35:33Z,OWNER,"To fix this I think I need to move the `load_template` implementation into a Jinja template loader. I'm not sure I'll be able to keep the `Templates considered` comment working though: https://github.com/simonw/datasette/blob/a2a709072059c6b3da365df9a332ca744c2079e9/datasette/app.py#L745-L750","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733499930,load_template hook doesn't work for include/extends, https://github.com/simonw/datasette/issues/1071#issuecomment-719777499,https://api.github.com/repos/simonw/datasette/issues/1071,719777499,MDEyOklzc3VlQ29tbWVudDcxOTc3NzQ5OQ==,9599,simonw,2020-10-30T20:20:01Z,2020-10-30T20:20:01Z,OWNER,"Fixed: https://latest.datasette.io/-/messages ![demo-message](https://user-images.githubusercontent.com/9599/97753199-9c142980-1ab2-11eb-9be9-c0be41acc68e.gif) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733485423,Messages should be displayed full width, https://github.com/simonw/datasette/pull/1069#issuecomment-719657478,https://api.github.com/repos/simonw/datasette/issues/1069,719657478,MDEyOklzc3VlQ29tbWVudDcxOTY1NzQ3OA==,22429695,codecov[bot],2020-10-30T16:31:21Z,2020-10-30T17:46:36Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1069?src=pr&el=h1) Report > Merging [#1069](https://codecov.io/gh/simonw/datasette/pull/1069?src=pr&el=desc) into [main](https://codecov.io/gh/simonw/datasette/commit/222f79bb4c6e2aa5426cc5ff25f1b2461e18a300?el=desc) will **increase** coverage by `0.01%`. > The diff coverage is `95.83%`. [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1069/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1)](https://codecov.io/gh/simonw/datasette/pull/1069?src=pr&el=tree) ```diff @@ Coverage Diff @@ ## main #1069 +/- ## ========================================== + Coverage 91.30% 91.32% +0.01% ========================================== Files 29 29 Lines 3736 3756 +20 ========================================== + Hits 3411 3430 +19 - Misses 325 326 +1 ``` | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1069?src=pr&el=tree) | Coverage Δ | | |---|---|---| | [datasette/views/base.py](https://codecov.io/gh/simonw/datasette/pull/1069/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3ZpZXdzL2Jhc2UucHk=) | `93.94% <ø> (-0.04%)` | :arrow_down: | | [datasette/app.py](https://codecov.io/gh/simonw/datasette/pull/1069/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `96.38% <95.45%> (-0.05%)` | :arrow_down: | | [datasette/hookspecs.py](https://codecov.io/gh/simonw/datasette/pull/1069/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2hvb2tzcGVjcy5weQ==) | `100.00% <100.00%> (ø)` | | ------ [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1069?src=pr&el=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1069?src=pr&el=footer). Last update [222f79b...92f3840](https://codecov.io/gh/simonw/datasette/pull/1069?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733303548,load_template() plugin hook, https://github.com/simonw/datasette/pull/1069#issuecomment-719672967,https://api.github.com/repos/simonw/datasette/issues/1069,719672967,MDEyOklzc3VlQ29tbWVudDcxOTY3Mjk2Nw==,9599,simonw,2020-10-30T16:58:01Z,2020-10-30T16:58:01Z,OWNER,"OK, new hook specification is: ```python @hookspec(firstresult=True) def load_template(template, request, datasette): ""Load the specified template, returning the template code as a string"" ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733303548,load_template() plugin hook, https://github.com/simonw/datasette/pull/1069#issuecomment-719670714,https://api.github.com/repos/simonw/datasette/issues/1069,719670714,MDEyOklzc3VlQ29tbWVudDcxOTY3MDcxNA==,9599,simonw,2020-10-30T16:53:56Z,2020-10-30T16:53:56Z,OWNER,"I'm having second thoughts about the design of the plugin hook. Consider the following: ```python plugin_template_source = pm.hook.load_template( template=template_name, database=context.get(""database""), table=context.get(""table""), columns=context.get(""columns""), view_name=self.name, request=request, datasette=self.ds, ) ``` It's a bit gross that `database`, `table` and `columns` are pulled out of the context like that. This doesn't make sense for pages that are rendered by plugins, for example. So maybe for the first release of this plugin hook I should cut it down to just seeing `template`, `request` and `datasette`. I can add the table/view/etc stuff back in later if it turns out to be necessary.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733303548,load_template() plugin hook, https://github.com/simonw/datasette/pull/1069#issuecomment-719666912,https://api.github.com/repos/simonw/datasette/issues/1069,719666912,MDEyOklzc3VlQ29tbWVudDcxOTY2NjkxMg==,9599,simonw,2020-10-30T16:47:44Z,2020-10-30T16:47:44Z,OWNER,"Bringing over a comment from #1042: > I'd like to do this all in the `datasette.render_template()` method to ensure it's available to plugins as well, not just core code that uses the `BaseView` class. > > This code is the problem: > > https://github.com/simonw/datasette/blob/d3e9b0aecb6f8e9b2befd9c654ccb7ce852db3e7/datasette/views/base.py#L114-L133 > > I think I'll fix this by moving the `select_templates` mechanism into `datasette.render_templates()`. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733303548,load_template() plugin hook, https://github.com/simonw/datasette/pull/1069#issuecomment-719664530,https://api.github.com/repos/simonw/datasette/issues/1069,719664530,MDEyOklzc3VlQ29tbWVudDcxOTY2NDUzMA==,9599,simonw,2020-10-30T16:43:40Z,2020-10-30T16:43:40Z,OWNER,I should include an example in the documentation that shows loading templates from a database table.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733303548,load_template() plugin hook, https://github.com/simonw/datasette/pull/1069#issuecomment-719640430,https://api.github.com/repos/simonw/datasette/issues/1069,719640430,MDEyOklzc3VlQ29tbWVudDcxOTY0MDQzMA==,9599,simonw,2020-10-30T16:01:13Z,2020-10-30T16:01:13Z,OWNER,Next steps: build a demonstration plugin against this.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",733303548,load_template() plugin hook, https://github.com/simonw/datasette/issues/1068#issuecomment-719630745,https://api.github.com/repos/simonw/datasette/issues/1068,719630745,MDEyOklzc3VlQ29tbWVudDcxOTYzMDc0NQ==,9599,simonw,2020-10-30T15:44:13Z,2020-10-30T15:44:13Z,OWNER,Documentation: https://docs.datasette.io/en/latest/authentication.html#debug-menu,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732939921,Default menu links should check a real permission , https://github.com/simonw/datasette/issues/1068#issuecomment-719332460,https://api.github.com/repos/simonw/datasette/issues/1068,719332460,MDEyOklzc3VlQ29tbWVudDcxOTMzMjQ2MA==,9599,simonw,2020-10-30T07:13:10Z,2020-10-30T07:13:10Z,OWNER,I mainly want this so I can add that debug menu to my Dogsheep.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732939921,Default menu links should check a real permission , https://github.com/simonw/datasette/issues/1068#issuecomment-719331236,https://api.github.com/repos/simonw/datasette/issues/1068,719331236,MDEyOklzc3VlQ29tbWVudDcxOTMzMTIzNg==,9599,simonw,2020-10-30T07:11:58Z,2020-10-30T07:11:58Z,OWNER,Document the new permission here: https://docs.datasette.io/en/stable/authentication.html#built-in-permissions,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732939921,Default menu links should check a real permission , https://github.com/simonw/datasette/issues/1068#issuecomment-719329219,https://api.github.com/repos/simonw/datasette/issues/1068,719329219,MDEyOklzc3VlQ29tbWVudDcxOTMyOTIxOQ==,9599,simonw,2020-10-30T07:09:59Z,2020-10-30T07:09:59Z,OWNER,Permission idea: `debug-menu`,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732939921,Default menu links should check a real permission , https://github.com/simonw/datasette/issues/1068#issuecomment-719328661,https://api.github.com/repos/simonw/datasette/issues/1068,719328661,MDEyOklzc3VlQ29tbWVudDcxOTMyODY2MQ==,9599,simonw,2020-10-30T07:09:30Z,2020-10-30T07:09:30Z,OWNER,Then this can make it available to root: https://github.com/simonw/datasette/blob/18a64fbb29271ce607937110bbdb55488c43f4e0/datasette/default_permissions.py,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732939921,Default menu links should check a real permission , https://github.com/simonw/datasette/issues/1067#issuecomment-719322666,https://api.github.com/repos/simonw/datasette/issues/1067,719322666,MDEyOklzc3VlQ29tbWVudDcxOTMyMjY2Ng==,9599,simonw,2020-10-30T07:04:02Z,2020-10-30T07:04:02Z,OWNER,"Maybe rename it to `actions_menu` and have it work for database, view, table and query pages using different arguments on each.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732905360,"Table actions menu on view pages, not on query pages", https://github.com/simonw/datasette/issues/1067#issuecomment-719320948,https://api.github.com/repos/simonw/datasette/issues/1067,719320948,MDEyOklzc3VlQ29tbWVudDcxOTMyMDk0OA==,9599,simonw,2020-10-30T07:02:37Z,2020-10-30T07:02:37Z,OWNER,"Yes, this should be possible - no point restricting what plugin authors can do with the feature. Will need to add some extra arguments to the plugin hook for this.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732905360,"Table actions menu on view pages, not on query pages", https://github.com/simonw/datasette/issues/690#issuecomment-719195346,https://api.github.com/repos/simonw/datasette/issues/690,719195346,MDEyOklzc3VlQ29tbWVudDcxOTE5NTM0Ng==,9599,simonw,2020-10-30T05:20:42Z,2020-10-30T05:20:42Z,OWNER,"I've now added two new plugin hooks: [menu_links()](https://docs.datasette.io/en/latest/plugin_hooks.html#menu-links-datasette-actor) and [table_actions()](https://docs.datasette.io/en/latest/plugin_hooks.html#table-actions-datasette-actor-database-table). I'm going to close this issue. Further work (on column actions and and database actions) can happen in separate tickets, but I won't include them in Datasette 0.51 since they're much less interesting than table and instance actions.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",573755726,Mechanism for plugins to add action menu items for various things, https://github.com/simonw/datasette/issues/1066#issuecomment-719194756,https://api.github.com/repos/simonw/datasette/issues/1066,719194756,MDEyOklzc3VlQ29tbWVudDcxOTE5NDc1Ng==,9599,simonw,2020-10-30T05:18:35Z,2020-10-30T05:18:35Z,OWNER,Documentation: https://docs.datasette.io/en/latest/plugin_hooks.html#table-actions-datasette-actor-database-table,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732859030,Table actions menu plus plugin hook, https://github.com/simonw/datasette/issues/1066#issuecomment-719194619,https://api.github.com/repos/simonw/datasette/issues/1066,719194619,MDEyOklzc3VlQ29tbWVudDcxOTE5NDYxOQ==,9599,simonw,2020-10-30T05:18:04Z,2020-10-30T05:18:04Z,OWNER,"The cog only appears if at least one table action has been registered by a plugin. It looks like this: ![table-actions](https://user-images.githubusercontent.com/9599/97662535-9bd54900-1a34-11eb-8e0f-56d159c8834e.gif) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732859030,Table actions menu plus plugin hook, https://github.com/simonw/datasette/issues/1066#issuecomment-719154646,https://api.github.com/repos/simonw/datasette/issues/1066,719154646,MDEyOklzc3VlQ29tbWVudDcxOTE1NDY0Ng==,9599,simonw,2020-10-30T03:48:15Z,2020-10-30T03:48:15Z,OWNER,This will use a very similar implementation to the navigation menu in #1064 - similar plugin hook and I'll use a `
` to implement it.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732859030,Table actions menu plus plugin hook, https://github.com/simonw/datasette/pull/1065#issuecomment-719153773,https://api.github.com/repos/simonw/datasette/issues/1065,719153773,MDEyOklzc3VlQ29tbWVudDcxOTE1Mzc3Mw==,22429695,codecov[bot],2020-10-30T03:44:57Z,2020-10-30T03:44:57Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1065?src=pr&el=h1) Report > Merging [#1065](https://codecov.io/gh/simonw/datasette/pull/1065?src=pr&el=desc) into [main](https://codecov.io/gh/simonw/datasette/commit/1a861be19e326e0c88230a711a1b6536366697d7?el=desc) will **increase** coverage by `0.03%`. > The diff coverage is `100.00%`. [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1065/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1)](https://codecov.io/gh/simonw/datasette/pull/1065?src=pr&el=tree) ```diff @@ Coverage Diff @@ ## main #1065 +/- ## ========================================== + Coverage 91.23% 91.27% +0.03% ========================================== Files 28 29 +1 Lines 3710 3724 +14 ========================================== + Hits 3385 3399 +14 Misses 325 325 ``` | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1065?src=pr&el=tree) | Coverage Δ | | |---|---|---| | [datasette/plugins.py](https://codecov.io/gh/simonw/datasette/pull/1065/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3BsdWdpbnMucHk=) | `82.35% <ø> (ø)` | | | [datasette/app.py](https://codecov.io/gh/simonw/datasette/pull/1065/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `96.42% <100.00%> (+0.03%)` | :arrow_up: | | [datasette/default\_menu\_links.py](https://codecov.io/gh/simonw/datasette/pull/1065/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2RlZmF1bHRfbWVudV9saW5rcy5weQ==) | `100.00% <100.00%> (ø)` | | | [datasette/hookspecs.py](https://codecov.io/gh/simonw/datasette/pull/1065/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2hvb2tzcGVjcy5weQ==) | `100.00% <100.00%> (ø)` | | ------ [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1065?src=pr&el=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1065?src=pr&el=footer). Last update [1a861be...5f118b5](https://codecov.io/gh/simonw/datasette/pull/1065?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732856937,Nav menu plus menu_links() hook, https://github.com/simonw/datasette/issues/1064#issuecomment-719117185,https://api.github.com/repos/simonw/datasette/issues/1064,719117185,MDEyOklzc3VlQ29tbWVudDcxOTExNzE4NQ==,9599,simonw,2020-10-30T01:35:17Z,2020-10-30T01:35:17Z,OWNER,"I'm going to go with a list of `{""label"": ..., ""href"": ...}` as the first iteration of this. The logout link will not be returned as part of the plugin output. A default plugin will provide the debug tools if the user is logged in as root.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732798913,Navigation menu plus plugin hook, https://github.com/simonw/datasette/issues/1064#issuecomment-719111597,https://api.github.com/repos/simonw/datasette/issues/1064,719111597,MDEyOklzc3VlQ29tbWVudDcxOTExMTU5Nw==,9599,simonw,2020-10-30T01:15:05Z,2020-10-30T01:15:05Z,OWNER,I'm torn on this one. I think I have a very slight preference for plugins returning structured objects as opposed to HTML. Less likely to regret that choice in the future?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732798913,Navigation menu plus plugin hook, https://github.com/simonw/datasette/issues/1064#issuecomment-719111373,https://api.github.com/repos/simonw/datasette/issues/1064,719111373,MDEyOklzc3VlQ29tbWVudDcxOTExMTM3Mw==,9599,simonw,2020-10-30T01:14:13Z,2020-10-30T01:14:13Z,OWNER,"Plugins returning HTML makes more sense for some of the other areas that plugins will be able to inject content - e.g. injecting content on the table or row page above the table. If I'm going to have that as a pattern though it may make sense to use HTML here, since that will be consistent with other places that plugins can inject additional content.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732798913,Navigation menu plus plugin hook, https://github.com/simonw/datasette/issues/1064#issuecomment-719110808,https://api.github.com/repos/simonw/datasette/issues/1064,719110808,MDEyOklzc3VlQ29tbWVudDcxOTExMDgwOA==,9599,simonw,2020-10-30T01:12:09Z,2020-10-30T01:12:19Z,OWNER,Or... plugins could return HTML - maybe optionally using helper functions to generate common HTML such that plugins which use the helpers can have their HTML modified in the future.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732798913,Navigation menu plus plugin hook, https://github.com/simonw/datasette/issues/1064#issuecomment-719110582,https://api.github.com/repos/simonw/datasette/issues/1064,719110582,MDEyOklzc3VlQ29tbWVudDcxOTExMDU4Mg==,9599,simonw,2020-10-30T01:11:13Z,2020-10-30T01:11:13Z,OWNER,"Should plugins be able to add forms like the logout form here, or should they be restricted to adding navigation links? I can't think of a reason a plugin would need to add a form. The logout form is a special case to protect against logout-csrf attacks. So I think plugins get to return a list of dictionaries, each with a `label` and an `href`: ```python return [{ ""label"": ""Upload CSVs"", ""href"": datasette.urls.path(""/-/upload-csvs"") }] ``` But... is there an argument for returning headings, to divide up the menu? I think so. I also like the idea that a default plugin checks for the `root` user and outputs links to the different debugging tools - maybe those should be wrapped in a section heading.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732798913,Navigation menu plus plugin hook, https://github.com/simonw/datasette/issues/1064#issuecomment-719109770,https://api.github.com/repos/simonw/datasette/issues/1064,719109770,MDEyOklzc3VlQ29tbWVudDcxOTEwOTc3MA==,9599,simonw,2020-10-30T01:08:14Z,2020-10-30T01:08:14Z,OWNER,"How should the plugin hook work? Here's the first version of the HTML: ```html
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732798913,Navigation menu plus plugin hook, https://github.com/simonw/datasette/issues/1064#issuecomment-719106174,https://api.github.com/repos/simonw/datasette/issues/1064,719106174,MDEyOklzc3VlQ29tbWVudDcxOTEwNjE3NA==,9599,simonw,2020-10-30T00:55:12Z,2020-10-30T00:55:12Z,OWNER,"So what should go in this menu? If the user is logged in as root, I'll link to the various debug pages. If they're not logged in at all I don't think the menu should appear. If they are logged in as anyone, it should display to give them access to the ""log out"" button. Plugins can add links to it. If those plugins add links, the menu will display.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732798913,Navigation menu plus plugin hook, https://github.com/simonw/datasette/issues/1064#issuecomment-719105641,https://api.github.com/repos/simonw/datasette/issues/1064,719105641,MDEyOklzc3VlQ29tbWVudDcxOTEwNTY0MQ==,9599,simonw,2020-10-30T00:53:00Z,2020-10-30T00:53:00Z,OWNER,Tips for making this accessible: https://css-tricks.com/accessible-svgs/,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732798913,Navigation menu plus plugin hook, https://github.com/simonw/datasette/issues/1064#issuecomment-719104883,https://api.github.com/repos/simonw/datasette/issues/1064,719104883,MDEyOklzc3VlQ29tbWVudDcxOTEwNDg4Mw==,9599,simonw,2020-10-30T00:50:01Z,2020-10-30T00:52:29Z,OWNER,"Here's what the prototype looks like so far: ![menu](https://user-images.githubusercontent.com/9599/97647443-7eda4f00-1a0f-11eb-8d78-b703b7a13616.gif) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732798913,Navigation menu plus plugin hook, https://github.com/simonw/datasette/issues/1064#issuecomment-719105197,https://api.github.com/repos/simonw/datasette/issues/1064,719105197,MDEyOklzc3VlQ29tbWVudDcxOTEwNTE5Nw==,9599,simonw,2020-10-30T00:51:16Z,2020-10-30T00:51:16Z,OWNER,"I used a `
` for this: https://github.com/simonw/datasette/blob/0d7ac764861d84be24d661cf4104ce61ea11a82a/datasette/templates/base.html#L16-L36 I added a bit of JavaScript so that clicking outside the menu would close it: https://github.com/simonw/datasette/blob/0d7ac764861d84be24d661cf4104ce61ea11a82a/datasette/templates/base.html#L59-L74","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732798913,Navigation menu plus plugin hook, https://github.com/simonw/datasette/issues/1034#issuecomment-719094027,https://api.github.com/repos/simonw/datasette/issues/1034,719094027,MDEyOklzc3VlQ29tbWVudDcxOTA5NDAyNw==,9599,simonw,2020-10-30T00:11:17Z,2020-10-30T00:11:17Z,OWNER,"Demos: https://latest.datasette.io/fixtures/binary_data.csv?_size=max ```csv rowid,data 1,http://latest.datasette.io/fixtures/binary_data/1.blob?_blob_column=data 2,http://latest.datasette.io/fixtures/binary_data/2.blob?_blob_column=data 3, ``` https://latest.datasette.io/fixtures.csv?sql=select+rowid%2C+data+from+binary_data+order+by+rowid+limit+1001&_size=max ```csv rowid,data 1,http://latest.datasette.io/fixtures.blob?sql=select+rowid%2C+data+from+binary_data+order+by+rowid+limit+1001&_size=max&_blob_column=data&_blob_hash=f3088978da8f9aea479ffc7f631370b968d2e855eeb172bea7f6c7a04262bb6d 2,http://latest.datasette.io/fixtures.blob?sql=select+rowid%2C+data+from+binary_data+order+by+rowid+limit+1001&_size=max&_blob_column=data&_blob_hash=b835b0483cedb86130b9a2c280880bf5fadc5318ddf8c18d0df5204d40df1724 3, ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725184645,Better way of representing binary data in .csv output, https://github.com/simonw/datasette/issues/1063#issuecomment-719066706,https://api.github.com/repos/simonw/datasette/issues/1063,719066706,MDEyOklzc3VlQ29tbWVudDcxOTA2NjcwNg==,9599,simonw,2020-10-29T22:46:28Z,2020-10-29T22:46:28Z,OWNER,I'm not going to do the base64 thing unless someone asks for it.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732685643,.csv should link to .blob downloads, https://github.com/simonw/datasette/issues/1051#issuecomment-719053669,https://api.github.com/repos/simonw/datasette/issues/1051,719053669,MDEyOklzc3VlQ29tbWVudDcxOTA1MzY2OQ==,9599,simonw,2020-10-29T22:12:16Z,2020-10-29T22:12:16Z,OWNER,"https://latest.datasette.io/fixtures?sql=select+rowid%2C+data+from+binary_data+order+by+rowid+limit+101 now looks like this: ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",729096595,Better display of binary data on arbitrary query results page, https://github.com/simonw/datasette/issues/1034#issuecomment-719050754,https://api.github.com/repos/simonw/datasette/issues/1034,719050754,MDEyOklzc3VlQ29tbWVudDcxOTA1MDc1NA==,9599,simonw,2020-10-29T22:04:52Z,2020-10-29T22:04:52Z,OWNER,I'm going to link to. the new `.blob` representation using the new `?_blob_hash=xxx` argument to ensure that the content served is the expected binary blob.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725184645,Better way of representing binary data in .csv output, https://github.com/simonw/datasette/issues/1063#issuecomment-719050390,https://api.github.com/repos/simonw/datasette/issues/1063,719050390,MDEyOklzc3VlQ29tbWVudDcxOTA1MDM5MA==,9599,simonw,2020-10-29T22:04:00Z,2020-10-29T22:04:00Z,OWNER,This will close #1034.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732685643,.csv should link to .blob downloads, https://github.com/simonw/datasette/pull/1061#issuecomment-719049115,https://api.github.com/repos/simonw/datasette/issues/1061,719049115,MDEyOklzc3VlQ29tbWVudDcxOTA0OTExNQ==,22429695,codecov[bot],2020-10-29T22:00:57Z,2020-10-29T22:00:57Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1061?src=pr&el=h1) Report > Merging [#1061](https://codecov.io/gh/simonw/datasette/pull/1061?src=pr&el=desc) into [main](https://codecov.io/gh/simonw/datasette/commit/d6f9ff71378c4eab34dad181c23cfc143a4aef2d?el=desc) will **increase** coverage by `0.07%`. > The diff coverage is `96.87%`. [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1061/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1)](https://codecov.io/gh/simonw/datasette/pull/1061?src=pr&el=tree) ```diff @@ Coverage Diff @@ ## main #1061 +/- ## ========================================== + Coverage 91.13% 91.20% +0.07% ========================================== Files 27 28 +1 Lines 3677 3697 +20 ========================================== + Hits 3351 3372 +21 + Misses 326 325 -1 ``` | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1061?src=pr&el=tree) | Coverage Δ | | |---|---|---| | [datasette/plugins.py](https://codecov.io/gh/simonw/datasette/pull/1061/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3BsdWdpbnMucHk=) | `82.35% <ø> (ø)` | | | [datasette/views/base.py](https://codecov.io/gh/simonw/datasette/pull/1061/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3ZpZXdzL2Jhc2UucHk=) | `93.77% <0.00%> (ø)` | | | [datasette/app.py](https://codecov.io/gh/simonw/datasette/pull/1061/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `96.38% <100.00%> (+0.15%)` | :arrow_up: | | [datasette/blob\_renderer.py](https://codecov.io/gh/simonw/datasette/pull/1061/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2Jsb2JfcmVuZGVyZXIucHk=) | `100.00% <100.00%> (ø)` | | | [datasette/utils/asgi.py](https://codecov.io/gh/simonw/datasette/pull/1061/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3V0aWxzL2FzZ2kucHk=) | `92.13% <100.00%> (+0.17%)` | :arrow_up: | | [datasette/views/database.py](https://codecov.io/gh/simonw/datasette/pull/1061/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3ZpZXdzL2RhdGFiYXNlLnB5) | `97.04% <100.00%> (+0.07%)` | :arrow_up: | | [datasette/views/table.py](https://codecov.io/gh/simonw/datasette/pull/1061/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3ZpZXdzL3RhYmxlLnB5) | `95.86% <100.00%> (-0.22%)` | :arrow_down: | ------ [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1061?src=pr&el=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1061?src=pr&el=footer). Last update [d6f9ff7...1196d08](https://codecov.io/gh/simonw/datasette/pull/1061?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732634375,.blob output renderer, https://github.com/simonw/datasette/pull/1061#issuecomment-719042601,https://api.github.com/repos/simonw/datasette/issues/1061,719042601,MDEyOklzc3VlQ29tbWVudDcxOTA0MjYwMQ==,9599,simonw,2020-10-29T21:45:35Z,2020-10-29T21:50:42Z,OWNER,Moving the CSV work to a separate issue.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732634375,.blob output renderer, https://github.com/simonw/datasette/issues/1063#issuecomment-719043108,https://api.github.com/repos/simonw/datasette/issues/1063,719043108,MDEyOklzc3VlQ29tbWVudDcxOTA0MzEwOA==,9599,simonw,2020-10-29T21:46:48Z,2020-10-29T21:46:48Z,OWNER,Remove this `xfail` and `import pytest`: https://github.com/simonw/datasette/blob/503a5b7b4080a26ef9ceb1ecd1a4a6f4ef4ffc59/tests/test_csv.py#L83-L96,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732685643,.csv should link to .blob downloads, https://github.com/simonw/datasette/pull/1061#issuecomment-719035336,https://api.github.com/repos/simonw/datasette/issues/1061,719035336,MDEyOklzc3VlQ29tbWVudDcxOTAzNTMzNg==,9599,simonw,2020-10-29T21:29:29Z,2020-10-29T21:29:29Z,OWNER,Those display_rows have already been processed by the `render_cell` plugin hook: https://github.com/simonw/datasette/blob/d6f9ff71378c4eab34dad181c23cfc143a4aef2d/datasette/views/database.py#L320-L346,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732634375,.blob output renderer, https://github.com/simonw/datasette/pull/1061#issuecomment-719033013,https://api.github.com/repos/simonw/datasette/issues/1061,719033013,MDEyOklzc3VlQ29tbWVudDcxOTAzMzAxMw==,9599,simonw,2020-10-29T21:27:14Z,2020-10-29T21:27:14Z,OWNER,"Next challenge: link to `.blob` downloads from https://latest.datasette.io/fixtures?sql=select+rowid%2C+data+from+binary_data This will be a bit tricky. Here's how that template works at the moment: https://github.com/simonw/datasette/blob/d6f9ff71378c4eab34dad181c23cfc143a4aef2d/datasette/templates/query.html#L69-L77","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732634375,.blob output renderer, https://github.com/simonw/datasette/issues/1062#issuecomment-719031901,https://api.github.com/repos/simonw/datasette/issues/1062,719031901,MDEyOklzc3VlQ29tbWVudDcxOTAzMTkwMQ==,9599,simonw,2020-10-29T21:25:54Z,2020-10-29T21:25:54Z,OWNER,Relevant code: https://github.com/simonw/datasette/blob/d6f9ff71378c4eab34dad181c23cfc143a4aef2d/datasette/views/base.py#L258-L345,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",732674148,Refactor .csv to be an output renderer - and teach register_output_renderer to stream all rows, https://github.com/simonw/datasette/issues/1050#issuecomment-719021514,https://api.github.com/repos/simonw/datasette/issues/1050,719021514,MDEyOklzc3VlQ29tbWVudDcxOTAyMTUxNA==,9599,simonw,2020-10-29T21:05:08Z,2020-10-29T21:05:08Z,OWNER,"Idea: what if Datasette had a custom SQLite function that could be used to generate URLs to the row-level BLOB download for a value? Then custom SQL query authors could use that function to link to the relevant content. This could be expanded to exposing other `datasette.urls` functionality as well.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",729057388,Switch to .blob render extension for BLOB downloads, https://github.com/simonw/datasette/issues/1050#issuecomment-719001701,https://api.github.com/repos/simonw/datasette/issues/1050,719001701,MDEyOklzc3VlQ29tbWVudDcxOTAwMTcwMQ==,9599,simonw,2020-10-29T20:26:44Z,2020-10-29T20:26:44Z,OWNER,I'll do the rest of the work on this in the pull request #1061.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",729057388,Switch to .blob render extension for BLOB downloads, https://github.com/simonw/datasette/issues/1050#issuecomment-718989895,https://api.github.com/repos/simonw/datasette/issues/1050,718989895,MDEyOklzc3VlQ29tbWVudDcxODk4OTg5NQ==,9599,simonw,2020-10-29T20:04:15Z,2020-10-29T20:04:15Z,OWNER,I'll use `hashlib.sha256` for these hashes.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",729057388,Switch to .blob render extension for BLOB downloads, https://github.com/simonw/datasette/issues/1050#issuecomment-718987852,https://api.github.com/repos/simonw/datasette/issues/1050,718987852,MDEyOklzc3VlQ29tbWVudDcxODk4Nzg1Mg==,9599,simonw,2020-10-29T20:00:32Z,2020-10-29T20:00:32Z,OWNER,"The reason I like the `?_blob_hash=` solution is that it feels really misleading to provide a link to ""download this binary"" which could conceivably download some other data.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",729057388,Switch to .blob render extension for BLOB downloads, https://github.com/simonw/datasette/issues/1050#issuecomment-718980944,https://api.github.com/repos/simonw/datasette/issues/1050,718980944,MDEyOklzc3VlQ29tbWVudDcxODk4MDk0NA==,9599,simonw,2020-10-29T19:46:19Z,2020-10-29T19:46:19Z,OWNER,"Had an idea in https://github.com/simonw/datasette/issues/1051#issuecomment-718980659 > OK, alternative idea. The `.blob` output renderer from #1050 gets to see multiple rows at once. > > For an arbitrary SQL query, how about if I link to this? > > `/db.blob?sql=...&_blob_column=data&_blob_hash=bc4c24181ed3ce666` > > Then the output renderer loops through all of the `data` results that are available to it and, if one of them hashes to that value, serves up that data? > > If no matches are found it can show an error message telling you that the link has expired (presumably because the underlying database has changed since the link was generated). > > I think this might be the best solution to the problem. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",729057388,Switch to .blob render extension for BLOB downloads, https://github.com/simonw/datasette/issues/1051#issuecomment-718980659,https://api.github.com/repos/simonw/datasette/issues/1051,718980659,MDEyOklzc3VlQ29tbWVudDcxODk4MDY1OQ==,9599,simonw,2020-10-29T19:45:42Z,2020-10-29T19:45:42Z,OWNER,"OK, alternative idea. The `.blob` output renderer from #1050 gets to see multiple rows at once. For an arbitrary SQL query, how about if I link to this? `/db.blob?sql=...&_blob_column=data&_blob_hash=bc4c24181ed3ce666` Then the output renderer loops through all of the `data` results that are available to it and, if one of them hashes to that value, serves up that data? If no matches are found it can show an error message telling you that the link has expired (presumably because the underlying database has changed since the link was generated). I think this might be the best solution to the problem.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",729096595,Better display of binary data on arbitrary query results page, https://github.com/simonw/datasette/issues/1053#issuecomment-718976679,https://api.github.com/repos/simonw/datasette/issues/1053,718976679,MDEyOklzc3VlQ29tbWVudDcxODk3NjY3OQ==,9599,simonw,2020-10-29T19:37:57Z,2020-10-29T19:37:57Z,OWNER,https://docs.datasette.io/en/latest/writing_plugins.html#designing-urls-for-your-plugin,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",729604838,Document recommendations for plugin authors to design URLs, https://github.com/simonw/datasette/pull/1049#issuecomment-718528252,https://api.github.com/repos/simonw/datasette/issues/1049,718528252,MDEyOklzc3VlQ29tbWVudDcxODUyODI1Mg==,82988,psychemedia,2020-10-29T09:20:34Z,2020-10-29T09:20:34Z,CONTRIBUTOR,That workaround is probably fine. I was trying to work out whether there might be other situations where a pre-external package load might be useful but couldn't offhand bring any other examples to mind. The static plugins option also looks interesting.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",729017519,Add template block prior to extra URL loaders, https://github.com/simonw/datasette/issues/1050#issuecomment-718346019,https://api.github.com/repos/simonw/datasette/issues/1050,718346019,MDEyOklzc3VlQ29tbWVudDcxODM0NjAxOQ==,9599,simonw,2020-10-29T04:05:07Z,2020-10-29T04:05:07Z,OWNER,"Yes, confirmed - this is a bug where if the `BLOB` column contains a `null` you get a nasty exception if you try to download it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",729057388,Switch to .blob render extension for BLOB downloads, https://github.com/simonw/datasette/issues/1050#issuecomment-718342036,https://api.github.com/repos/simonw/datasette/issues/1050,718342036,MDEyOklzc3VlQ29tbWVudDcxODM0MjAzNg==,9599,simonw,2020-10-29T03:49:57Z,2020-10-29T03:49:57Z,OWNER,"@thadk from that error it looks like the problem may have been that you had a BLOB column containing a `null` value? If so that's definitely a bug, I'll fix that.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",729057388,Switch to .blob render extension for BLOB downloads, https://github.com/simonw/datasette/pull/1049#issuecomment-718340847,https://api.github.com/repos/simonw/datasette/issues/1049,718340847,MDEyOklzc3VlQ29tbWVudDcxODM0MDg0Nw==,9599,simonw,2020-10-29T03:45:47Z,2020-10-29T03:48:26Z,OWNER,"[thebe](https://thebelab.readthedocs.io/en/latest/examples/minimal_example.html) is the first time I've seen a library that requires you to set up some global JavaScript configuration before loading the script itself. I'm hesitant to add an extra template block just to cover that one case since it's such a rare pattern. But it's important that `thebelab` can be used with Datasette. Would this pattern work for you instead? ```html+jinja {% block extra_head %} {% endblock %} ``` ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",729017519,Add template block prior to extra URL loaders, https://github.com/simonw/datasette/pull/1049#issuecomment-718341542,https://api.github.com/repos/simonw/datasette/issues/1049,718341542,MDEyOklzc3VlQ29tbWVudDcxODM0MTU0Mg==,9599,simonw,2020-10-29T03:48:12Z,2020-10-29T03:48:12Z,OWNER,"You could use Datasette's new `{{ urls.static_plugins(...) }}` template option - see https://docs.datasette.io/en/latest/internals.html#internals-datasette-urls - to generate a link to code that was bundled with the plugin: ```html+jinja {% block extra_head %} {% endblock %} ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",729017519,Add template block prior to extra URL loaders, https://github.com/simonw/datasette/issues/1050#issuecomment-718317997,https://api.github.com/repos/simonw/datasette/issues/1050,718317997,MDEyOklzc3VlQ29tbWVudDcxODMxNzk5Nw==,283343,thadk,2020-10-29T02:24:50Z,2020-10-29T02:29:24Z,NONE,"Unsolicited feedback for an unreleased feature of the [current](https://github.com/simonw/datasette/commit/5e0b72247ecab4ce0fcec599b77a83d73a480872) unreleased GitHub version (I casually wanted to access a blob row) – the existing #1036 route doesn't support special characters in database or table names (e.g. `@()` ). Maybe this is motivation for your new idea here. Also I got this error/crash with my blob and wasn't able to get the file: https://gist.github.com/thadk/28ac32af0e88747ce9056c90b0b19d34","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",729057388,Switch to .blob render extension for BLOB downloads, https://github.com/simonw/datasette/pull/1060#issuecomment-718243062,https://api.github.com/repos/simonw/datasette/issues/1060,718243062,MDEyOklzc3VlQ29tbWVudDcxODI0MzA2Mg==,22429695,codecov[bot],2020-10-28T22:23:33Z,2020-10-28T22:23:33Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1060?src=pr&el=h1) Report > Merging [#1060](https://codecov.io/gh/simonw/datasette/pull/1060?src=pr&el=desc) into [main](https://codecov.io/gh/simonw/datasette/commit/abcf0222496d8148b2e585ffa0ff192270a04b06?el=desc) will **increase** coverage by `6.42%`. > The diff coverage is `100.00%`. [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1060/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1)](https://codecov.io/gh/simonw/datasette/pull/1060?src=pr&el=tree) ```diff @@ Coverage Diff @@ ## main #1060 +/- ## ========================================== + Coverage 84.71% 91.13% +6.42% ========================================== Files 28 27 -1 Lines 3957 3677 -280 ========================================== - Hits 3352 3351 -1 + Misses 605 326 -279 ``` | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1060?src=pr&el=tree) | Coverage Δ | | |---|---|---| | [datasette/cli.py](https://codecov.io/gh/simonw/datasette/pull/1060/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2NsaS5weQ==) | `73.63% <100.00%> (+0.13%)` | :arrow_up: | | [datasette/version.py](https://codecov.io/gh/simonw/datasette/pull/1060/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3ZlcnNpb24ucHk=) | `100.00% <100.00%> (ø)` | | ------ [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1060?src=pr&el=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1060?src=pr&el=footer). Last update [abcf022...4725d46](https://codecov.io/gh/simonw/datasette/pull/1060?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",731827081,New explicit versioning mechanism, https://github.com/simonw/sqlite-utils/issues/191#issuecomment-718170295,https://api.github.com/repos/simonw/sqlite-utils/issues/191,718170295,MDEyOklzc3VlQ29tbWVudDcxODE3MDI5NQ==,9599,simonw,2020-10-28T19:50:16Z,2020-10-28T19:50:16Z,OWNER,"I think I made a mistake when I designed the initial decorator. I should have had it work like this: ```python @db.register_function() def reverse_string(s): return """".join(reversed(list(s))) ``` As this leaves open the option to add new parameters in the future. To avoid breaking backwards compatibility I'll use the hack that detects the argument this time, but in the future I'll try to remember to always design decorators to be called like `@decorator()`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",731740458,Idea: @db.register_function(deterministic=True), https://github.com/simonw/sqlite-utils/issues/191#issuecomment-718168730,https://api.github.com/repos/simonw/sqlite-utils/issues/191,718168730,MDEyOklzc3VlQ29tbWVudDcxODE2ODczMA==,9599,simonw,2020-10-28T19:47:20Z,2020-10-28T19:47:20Z,OWNER,"https://stackoverflow.com/a/3931903 looks useful: ```python def trace(*args): def _trace(func): def wrapper(*args, **kwargs): print enter_string func(*args, **kwargs) print exit_string return wrapper if len(args) == 1 and callable(args[0]): # No arguments, this is the decorator # Set default values for the arguments enter_string = 'entering' exit_string = 'exiting' return _trace(args[0]) else: # This is just returning the decorator enter_string, exit_string = args return _trace ``` Can improve that code with `functools.wraps`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",731740458,Idea: @db.register_function(deterministic=True), https://github.com/simonw/datasette/pull/1059#issuecomment-718078447,https://api.github.com/repos/simonw/datasette/issues/1059,718078447,MDEyOklzc3VlQ29tbWVudDcxODA3ODQ0Nw==,9599,simonw,2020-10-28T17:07:59Z,2020-10-28T17:08:14Z,OWNER,"> #### 0.6.0 (2020-10-27) > > - aiofiles is now tested on ppc64le. > - Added name and mode properties to async file objects. [#82](https://github.com/Tinche/aiofiles/pull/82) > - Fixed a DeprecationWarning internally. [#75](https://github.com/Tinche/aiofiles/pull/75) > - Python 3.9 support and tests.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",731445447,"Update aiofiles requirement from <0.6,>=0.4 to >=0.4,<0.7", https://github.com/simonw/datasette/pull/1059#issuecomment-717938992,https://api.github.com/repos/simonw/datasette/issues/1059,717938992,MDEyOklzc3VlQ29tbWVudDcxNzkzODk5Mg==,22429695,codecov[bot],2020-10-28T13:38:46Z,2020-10-28T13:38:46Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1059?src=pr&el=h1) Report > Merging [#1059](https://codecov.io/gh/simonw/datasette/pull/1059?src=pr&el=desc) into [main](https://codecov.io/gh/simonw/datasette/commit/7d9fedc176717a7e3d22a96575ae0aada5a65440?el=desc) will **not change** coverage. > The diff coverage is `n/a`. [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1059/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1)](https://codecov.io/gh/simonw/datasette/pull/1059?src=pr&el=tree) ```diff @@ Coverage Diff @@ ## main #1059 +/- ## ======================================= Coverage 84.71% 84.71% ======================================= Files 28 28 Lines 3957 3957 ======================================= Hits 3352 3352 Misses 605 605 ``` ------ [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1059?src=pr&el=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1059?src=pr&el=footer). Last update [7d9fedc...e46327a](https://codecov.io/gh/simonw/datasette/pull/1059?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",731445447,"Update aiofiles requirement from <0.6,>=0.4 to >=0.4,<0.7", https://github.com/simonw/datasette/issues/1057#issuecomment-717531272,https://api.github.com/repos/simonw/datasette/issues/1057,717531272,MDEyOklzc3VlQ29tbWVudDcxNzUzMTI3Mg==,9599,simonw,2020-10-27T20:51:09Z,2020-10-27T20:51:09Z,OWNER,"That works! ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",730797787,--cors should enable /fixtures.db CORS access, https://github.com/simonw/datasette/issues/1058#issuecomment-717527606,https://api.github.com/repos/simonw/datasette/issues/1058,717527606,MDEyOklzc3VlQ29tbWVudDcxNzUyNzYwNg==,9599,simonw,2020-10-27T20:44:06Z,2020-10-27T20:44:06Z,OWNER,Example: https://github.com/simonw/datasette/blob/5a1519796037105bc20bcf2f91a76e022926c204/datasette/views/database.py#L26-L32,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",730802994,Database download should implement cascading permissions, https://github.com/simonw/datasette/pull/1056#issuecomment-717489501,https://api.github.com/repos/simonw/datasette/issues/1056,717489501,MDEyOklzc3VlQ29tbWVudDcxNzQ4OTUwMQ==,22429695,codecov[bot],2020-10-27T19:39:41Z,2020-10-27T19:39:41Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1056?src=pr&el=h1) Report > Merging [#1056](https://codecov.io/gh/simonw/datasette/pull/1056?src=pr&el=desc) into [main](https://codecov.io/gh/simonw/datasette/commit/26bb4a268127da2c38f4241abe45444b2a6f7874?el=desc) will **not change** coverage. > The diff coverage is `n/a`. [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1056/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1)](https://codecov.io/gh/simonw/datasette/pull/1056?src=pr&el=tree) ```diff @@ Coverage Diff @@ ## main #1056 +/- ## ======================================= Coverage 84.70% 84.70% ======================================= Files 28 28 Lines 3955 3955 ======================================= Hits 3350 3350 Misses 605 605 ``` ------ [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1056?src=pr&el=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1056?src=pr&el=footer). Last update [26bb4a2...a7b2aab](https://codecov.io/gh/simonw/datasette/pull/1056?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",730752399,"Radical new colour scheme and base styles, courtesy of @natbat", https://github.com/simonw/sqlite-utils/pull/189#issuecomment-717361487,https://api.github.com/repos/simonw/sqlite-utils/issues/189,717361487,MDEyOklzc3VlQ29tbWVudDcxNzM2MTQ4Nw==,9599,simonw,2020-10-27T16:24:04Z,2020-10-27T16:24:04Z,OWNER,"This is great, thank you very much.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",729818242,Allow iterables other than Lists in m2m records, https://github.com/simonw/sqlite-utils/pull/189#issuecomment-717359145,https://api.github.com/repos/simonw/sqlite-utils/issues/189,717359145,MDEyOklzc3VlQ29tbWVudDcxNzM1OTE0NQ==,35681,adamwolf,2020-10-27T16:20:32Z,2020-10-27T16:20:32Z,CONTRIBUTOR,"No problem. I added a test. Let me know if it looks sufficient or if you want me to to tweak something! If you don't mind, would you tag this PR as ""hacktoberfest-accepted""? If you do mind, no problem and I'm sorry for asking :) My kiddos like the shirts.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",729818242,Allow iterables other than Lists in m2m records, https://github.com/simonw/datasette/issues/1054#issuecomment-717051707,https://api.github.com/repos/simonw/datasette/issues/1054,717051707,MDEyOklzc3VlQ29tbWVudDcxNzA1MTcwNw==,9599,simonw,2020-10-27T07:41:21Z,2020-10-27T07:41:21Z,OWNER,Essentially it's this problem: https://github.com/python-versioneer/python-versioneer/issues/140,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",730199464,Switch from versioneer to concrete version in setup.py, https://github.com/simonw/datasette/issues/1054#issuecomment-717050585,https://api.github.com/repos/simonw/datasette/issues/1054,717050585,MDEyOklzc3VlQ29tbWVudDcxNzA1MDU4NQ==,9599,simonw,2020-10-27T07:38:50Z,2020-10-27T07:38:50Z,OWNER,"Maybe imitate how Django does this, e.g. https://github.com/django/django/commit/6b9b2af7352908d40ca4d31bdb1b80c013cab29a","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",730199464,Switch from versioneer to concrete version in setup.py, https://github.com/simonw/sqlite-utils/pull/189#issuecomment-716756103,https://api.github.com/repos/simonw/sqlite-utils/issues/189,716756103,MDEyOklzc3VlQ29tbWVudDcxNjc1NjEwMw==,9599,simonw,2020-10-26T18:56:19Z,2020-10-26T18:56:19Z,OWNER,"This is a great fix, thanks! If you add a unit test somewhere in here I'll merge the PR: https://github.com/simonw/sqlite-utils/blob/main/tests/test_m2m.py","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",729818242,Allow iterables other than Lists in m2m records, https://github.com/simonw/datasette/issues/1051#issuecomment-716681602,https://api.github.com/repos/simonw/datasette/issues/1051,716681602,MDEyOklzc3VlQ29tbWVudDcxNjY4MTYwMg==,9599,simonw,2020-10-26T16:51:58Z,2020-10-26T16:51:58Z,OWNER,"I still need to improve the current binary display on the query page though, where it outputs a Python `b'...'` literal.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",729096595,Better display of binary data on arbitrary query results page, https://github.com/simonw/datasette/issues/1051#issuecomment-716681167,https://api.github.com/repos/simonw/datasette/issues/1051,716681167,MDEyOklzc3VlQ29tbWVudDcxNjY4MTE2Nw==,9599,simonw,2020-10-26T16:51:15Z,2020-10-26T16:51:15Z,OWNER,"Crazy idea: generate a signed URL containing a base64 of the gzip of the binary content (to try and reduce size). No: this will blow through URL limits in various hosting providers and possibly even browsers. It could be made to work a little bit more reliably with some extra JavaScript that turns it into a download on the browser-side, but that would be hideously complicated. Also the signed bit doesn't prevent people from generating SQL queries that generate nasty binary blobs for download. I'm beginning to think that restricting this feature to just table view, not query view, is a better idea. Query view can still get at the binary using JSON and base64.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",729096595,Better display of binary data on arbitrary query results page, https://github.com/simonw/datasette/issues/976#issuecomment-716305890,https://api.github.com/repos/simonw/datasette/issues/976,716305890,MDEyOklzc3VlQ29tbWVudDcxNjMwNTg5MA==,9599,simonw,2020-10-26T05:07:10Z,2020-10-26T05:07:10Z,OWNER,"I used the new `datasette.urls` methods to handle escaping table names. https://github.com/simonw/datasette/blob/f5dbe61a4568c0915ec6be820095c2960cf0857c/datasette/utils/__init__.py#L996-L1008","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",708289783,Idea: -o could open to a more convenient location, https://github.com/simonw/datasette/issues/1052#issuecomment-716265360,https://api.github.com/repos/simonw/datasette/issues/1052,716265360,MDEyOklzc3VlQ29tbWVudDcxNjI2NTM2MA==,9599,simonw,2020-10-26T02:17:58Z,2020-10-26T02:17:58Z,OWNER,"The default z-index values for Leaflet are defined here: https://github.com/Leaflet/Leaflet/blob/b346bb8bf7bb80899baa1f4fc1536bae58e7e3e6/dist/leaflet.css#L81-L91 ```css .leaflet-pane { z-index: 400; } .leaflet-tile-pane { z-index: 200; } .leaflet-overlay-pane { z-index: 400; } .leaflet-shadow-pane { z-index: 500; } .leaflet-marker-pane { z-index: 600; } .leaflet-tooltip-pane { z-index: 650; } .leaflet-popup-pane { z-index: 700; } .leaflet-map-pane canvas { z-index: 100; } .leaflet-map-pane svg { z-index: 200; } ``` So a `z-index` of 1000 on the menu should fix this.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",729183332,Column action menu overlapped by Leaflet maps, https://github.com/simonw/datasette/pull/1043#issuecomment-716237524,https://api.github.com/repos/simonw/datasette/issues/1043,716237524,MDEyOklzc3VlQ29tbWVudDcxNjIzNzUyNA==,45380,bollwyvl,2020-10-26T00:14:57Z,2020-10-26T00:14:57Z,CONTRIBUTOR,"Sorry, I was out of the loop this weekend. The missing sdists were in some the `datasette-*` plugins... i'll capture my findings more concretely in one spot when i have a chance...","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",727915394,Include LICENSE in sdist, https://github.com/simonw/datasette/issues/1051#issuecomment-716204271,https://api.github.com/repos/simonw/datasette/issues/1051,716204271,MDEyOklzc3VlQ29tbWVudDcxNjIwNDI3MQ==,9599,simonw,2020-10-25T20:08:04Z,2020-10-25T20:08:04Z,OWNER,"This is bad though, because if I want to provide binary data in CSV as requested in #1034 I need some way of providing that data. Which suggests to me that the base64 option is the only one that can make sense for arbitrary SQL queries represented as CSV. Download links won't work.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",729096595,Better display of binary data on arbitrary query results page, https://github.com/simonw/datasette/issues/1051#issuecomment-716204090,https://api.github.com/repos/simonw/datasette/issues/1051,716204090,MDEyOklzc3VlQ29tbWVudDcxNjIwNDA5MA==,9599,simonw,2020-10-25T20:06:42Z,2020-10-25T20:06:42Z,OWNER,"Providing a binary download link here is actually extremely difficult. The problem is that the SQL query itself represents data that can change from one moment to the next. It's no good showing a ""Binary: 55 bytes"" message that links to that same SQL query but with a `.blob` extension and arguments to select the particular result, because the data may change in a way that causes that query to return a different row - at which point the download link will give you the wrong data, not the 55 bytes you asked for. So providing a download link risks being misleading.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",729096595,Better display of binary data on arbitrary query results page, https://github.com/simonw/datasette/issues/1050#issuecomment-716174203,https://api.github.com/repos/simonw/datasette/issues/1050,716174203,MDEyOklzc3VlQ29tbWVudDcxNjE3NDIwMw==,9599,simonw,2020-10-25T16:27:39Z,2020-10-25T16:53:27Z,OWNER,"Idea: `.blob` output rendererer, where you tell it which column you want using `?_blob_column=x`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",729057388,Switch to .blob render extension for BLOB downloads, https://github.com/simonw/datasette/issues/1050#issuecomment-716175236,https://api.github.com/repos/simonw/datasette/issues/1050,716175236,MDEyOklzc3VlQ29tbWVudDcxNjE3NTIzNg==,9599,simonw,2020-10-25T16:35:20Z,2020-10-25T16:35:20Z,OWNER,"This is clearly a better solution than the one I implemented in #1040 - I don't have to add a new route, I don't have to implement permission checks, it reuses mechanism.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",729057388,Switch to .blob render extension for BLOB downloads, https://github.com/simonw/datasette/pull/1049#issuecomment-716146238,https://api.github.com/repos/simonw/datasette/issues/1049,716146238,MDEyOklzc3VlQ29tbWVudDcxNjE0NjIzOA==,22429695,codecov[bot],2020-10-25T13:13:32Z,2020-10-25T13:13:32Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1049?src=pr&el=h1) Report > Merging [#1049](https://codecov.io/gh/simonw/datasette/pull/1049?src=pr&el=desc) into [main](https://codecov.io/gh/simonw/datasette/commit/42f4851e3e7885f1092f104d6c883cea40b12f02?el=desc) will **not change** coverage. > The diff coverage is `n/a`. [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1049/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1)](https://codecov.io/gh/simonw/datasette/pull/1049?src=pr&el=tree) ```diff @@ Coverage Diff @@ ## main #1049 +/- ## ======================================= Coverage 84.72% 84.72% ======================================= Files 28 28 Lines 3942 3942 ======================================= Hits 3340 3340 Misses 602 602 ``` ------ [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1049?src=pr&el=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1049?src=pr&el=footer). Last update [42f4851...50a743a](https://codecov.io/gh/simonw/datasette/pull/1049?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",729017519,Add template block prior to extra URL loaders, https://github.com/simonw/datasette/issues/838#issuecomment-716123598,https://api.github.com/repos/simonw/datasette/issues/838,716123598,MDEyOklzc3VlQ29tbWVudDcxNjEyMzU5OA==,82988,psychemedia,2020-10-25T10:20:12Z,2020-10-25T10:53:24Z,CONTRIBUTOR,"I'm trying to [run something behind a MyBinder proxy](https://github.com/ouseful-testing/nbsearch), but seem to have something set up incorrectly and not sure what the fix is? I'm starting datasette with jupyter-server-proxy setup: ``` # __init__.py def setup_nbsearch(): return { ""command"": [ ""datasette"", ""serve"", f""{_NBSEARCH_DB_PATH}"", ""-p"", ""{port}"", ""--config"", ""base_url:{base_url}nbsearch/"" ], ""absolute_url"": True, # The following needs a the labextension installing. # eg in postBuild: jupyter labextension install jupyterlab-server-proxy ""launcher_entry"": { ""enabled"": True, ""title"": ""nbsearch"", }, } ``` where the `base_url` gets automatically populated by the server-proxy. I define the loaders as: ``` # __init__.py from datasette import hookimpl @hookimpl def extra_css_urls(database, table, columns, view_name, datasette): return [ ""/-/static-plugins/nbsearch/prism.css"", ""/-/static-plugins/nbsearch/nbsearch.css"", ] ``` but these seem to also need a base_url prefix set somehow? Currently, the generated HTML loads properly but internal links are incorrect; eg they take the form `` which resolves to eg `https://notebooks.gesis.org/hub/-/static-plugins/nbsearch/prism.css` rather than required URL of form `https://notebooks.gesis.org/binder/jupyter/user/ouseful-testing-nbsearch-0fx1mx67/nbsearch/-/static-plugins/nbsearch/prism.css`. The main css is loaded correctly: ``","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",637395097,Incorrect URLs when served behind a proxy with base_url set, https://github.com/simonw/datasette/issues/1034#issuecomment-716078777,https://api.github.com/repos/simonw/datasette/issues/1034,716078777,MDEyOklzc3VlQ29tbWVudDcxNjA3ODc3Nw==,9599,simonw,2020-10-25T01:25:11Z,2020-10-25T01:25:11Z,OWNER,"SQLite actually has APIs that could help here: https://www.sqlite.org/c3ref/column_database_name.html - for any given SQL query they identify the origin/table/column that is the source of each resulting column. Those aren't exposed in the Python `sqlite3` module though, so using them could be extremely tricky.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725184645,Better way of representing binary data in .csv output, https://github.com/simonw/datasette/issues/1034#issuecomment-716078605,https://api.github.com/repos/simonw/datasette/issues/1034,716078605,MDEyOklzc3VlQ29tbWVudDcxNjA3ODYwNQ==,9599,simonw,2020-10-25T01:22:22Z,2020-10-25T01:22:22Z,OWNER,For arbitrary CSV the only solution I can think of is to embed the base64 value.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725184645,Better way of representing binary data in .csv output, https://github.com/simonw/datasette/issues/1034#issuecomment-716078512,https://api.github.com/repos/simonw/datasette/issues/1034,716078512,MDEyOklzc3VlQ29tbWVudDcxNjA3ODUxMg==,9599,simonw,2020-10-25T01:21:11Z,2020-10-25T01:21:11Z,OWNER,"What should happen for CSV export of arbitrary SQL queries, where there's no obvious BLOB to link to?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725184645,Better way of representing binary data in .csv output, https://github.com/simonw/datasette/issues/1034#issuecomment-716078420,https://api.github.com/repos/simonw/datasette/issues/1034,716078420,MDEyOklzc3VlQ29tbWVudDcxNjA3ODQyMA==,9599,simonw,2020-10-25T01:20:00Z,2020-10-25T01:20:00Z,OWNER,That documentation: https://docs.datasette.io/en/latest/internals.html#absolute-url-request-path,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725184645,Better way of representing binary data in .csv output, https://github.com/simonw/datasette/issues/1034#issuecomment-716077541,https://api.github.com/repos/simonw/datasette/issues/1034,716077541,MDEyOklzc3VlQ29tbWVudDcxNjA3NzU0MQ==,9599,simonw,2020-10-25T01:09:38Z,2020-10-25T01:10:04Z,OWNER,I should turn `datasette.absolute_url(...)` into a documented internal API on https://docs.datasette.io/en/stable/internals.html#datasette-class,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725184645,Better way of representing binary data in .csv output, https://github.com/simonw/datasette/issues/1034#issuecomment-716077508,https://api.github.com/repos/simonw/datasette/issues/1034,716077508,MDEyOklzc3VlQ29tbWVudDcxNjA3NzUwOA==,9599,simonw,2020-10-25T01:09:17Z,2020-10-25T01:09:17Z,OWNER,Here's how those absolute `next_url` values are generated: https://github.com/simonw/datasette/blob/5db7ae3ce165ded57c7fb1cfbdb3258b1cf06c10/datasette/views/table.py#L774-L776,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725184645,Better way of representing binary data in .csv output, https://github.com/simonw/datasette/issues/1034#issuecomment-716077436,https://api.github.com/repos/simonw/datasette/issues/1034,716077436,MDEyOklzc3VlQ29tbWVudDcxNjA3NzQzNg==,9599,simonw,2020-10-25T01:08:35Z,2020-10-25T01:08:42Z,OWNER,"This is actually a bit tricky to implement, for a few reasons: - Need to generate a full URL, including the `https://host/` bit. I've done this for `next_url` in the JSON output before, thankfully. - This only makes sense for CSV output for tables. If it's the CSV output of an arbitrary query there's no `/db/table/-/blob/pk/column.blob` page for me to link to. - Need to generate those `/.../-/blob/...` URLs for the data that is being output as CSV.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725184645,Better way of representing binary data in .csv output, https://github.com/simonw/datasette/issues/1034#issuecomment-713277810,https://api.github.com/repos/simonw/datasette/issues/1034,713277810,MDEyOklzc3VlQ29tbWVudDcxMzI3NzgxMA==,9599,simonw,2020-10-21T03:40:50Z,2020-10-25T01:01:23Z,OWNER,Blocked awaiting #1036 (update: now unblocked),"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725184645,Better way of representing binary data in .csv output, https://github.com/simonw/datasette/issues/1046#issuecomment-716071507,https://api.github.com/repos/simonw/datasette/issues/1046,716071507,MDEyOklzc3VlQ29tbWVudDcxNjA3MTUwNw==,9599,simonw,2020-10-25T00:06:47Z,2020-10-25T00:06:47Z,OWNER,"I used https://primer.style/octicons/download-16 instead. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",728895193,Link to blob downloads in the right places, https://github.com/simonw/datasette/pull/1040#issuecomment-713920562,https://api.github.com/repos/simonw/datasette/issues/1040,713920562,MDEyOklzc3VlQ29tbWVudDcxMzkyMDU2Mg==,22429695,codecov[bot],2020-10-21T22:44:12Z,2020-10-24T23:08:14Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1040?src=pr&el=h1) Report > Merging [#1040](https://codecov.io/gh/simonw/datasette/pull/1040?src=pr&el=desc) into [main](https://codecov.io/gh/simonw/datasette/commit/bf82b3d6a605c9ddadd5fb739249dfe6defaf635?el=desc) will **increase** coverage by `0.10%`. > The diff coverage is `100.00%`. [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1040/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1)](https://codecov.io/gh/simonw/datasette/pull/1040?src=pr&el=tree) ```diff @@ Coverage Diff @@ ## main #1040 +/- ## ========================================== + Coverage 84.65% 84.76% +0.10% ========================================== Files 28 28 Lines 3924 3938 +14 ========================================== + Hits 3322 3338 +16 + Misses 602 600 -2 ``` | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1040?src=pr&el=tree) | Coverage Δ | | |---|---|---| | [datasette/views/index.py](https://codecov.io/gh/simonw/datasette/pull/1040/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3ZpZXdzL2luZGV4LnB5) | `98.18% <ø> (+1.69%)` | :arrow_up: | | [datasette/views/special.py](https://codecov.io/gh/simonw/datasette/pull/1040/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3ZpZXdzL3NwZWNpYWwucHk=) | `92.70% <ø> (-0.82%)` | :arrow_down: | | [datasette/app.py](https://codecov.io/gh/simonw/datasette/pull/1040/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `96.37% <100.00%> (+0.17%)` | :arrow_up: | | [datasette/views/base.py](https://codecov.io/gh/simonw/datasette/pull/1040/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3ZpZXdzL2Jhc2UucHk=) | `93.77% <100.00%> (ø)` | | | [datasette/views/table.py](https://codecov.io/gh/simonw/datasette/pull/1040/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3ZpZXdzL3RhYmxlLnB5) | `96.07% <100.00%> (+0.22%)` | :arrow_up: | ------ [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1040?src=pr&el=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1040?src=pr&el=footer). Last update [bf82b3d...4f3165f](https://codecov.io/gh/simonw/datasette/pull/1040?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",726910999,/db/table/-/blob/pk/column.blob download URL, https://github.com/simonw/datasette/issues/1046#issuecomment-716066342,https://api.github.com/repos/simonw/datasette/issues/1046,716066342,MDEyOklzc3VlQ29tbWVudDcxNjA2NjM0Mg==,9599,simonw,2020-10-24T23:02:07Z,2020-10-24T23:02:25Z,OWNER,"A download icon would be nice for the links in the table display. I like this one https://primer.style/octicons/download-24 ```svg ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",728895193,Link to blob downloads in the right places, https://github.com/simonw/datasette/issues/1033#issuecomment-716066000,https://api.github.com/repos/simonw/datasette/issues/1033,716066000,MDEyOklzc3VlQ29tbWVudDcxNjA2NjAwMA==,82988,psychemedia,2020-10-24T22:58:33Z,2020-10-24T22:58:33Z,CONTRIBUTOR,"From [the docs](https://docs.datasette.io/en/latest/internals.html#datasette-urls), I note: ``` datasette.urls.instance() Returns the URL to the Datasette instance root page. This is usually ""/"" ``` What about the proxy case? Eg if I am using jupyter-server-proxy on a MyBinder or local Jupyter notebook server site, `https://example.com:PORT/weirdpath/datasette`, what does `datasette.urls.instance()` refer to? - [ ] `https://example.com:PORT/weirdpath/datasette` - [ ] `https://example.com:PORT/weirdpath/` - [ ] `https://example.com:PORT/` - [ ] `https://example.com` - [ ] something else?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725099777,datasette.urls.static_plugins(...) method, https://github.com/simonw/datasette/issues/1033#issuecomment-716048564,https://api.github.com/repos/simonw/datasette/issues/1033,716048564,MDEyOklzc3VlQ29tbWVudDcxNjA0ODU2NA==,9599,simonw,2020-10-24T20:08:31Z,2020-10-24T20:08:31Z,OWNER,Documentation here: https://docs.datasette.io/en/latest/internals.html#datasette-urls,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725099777,datasette.urls.static_plugins(...) method, https://github.com/simonw/datasette/issues/575#issuecomment-716048199,https://api.github.com/repos/simonw/datasette/issues/575,716048199,MDEyOklzc3VlQ29tbWVudDcxNjA0ODE5OQ==,9599,simonw,2020-10-24T20:05:44Z,2020-10-24T20:05:44Z,OWNER,https://docs.datasette.io/en/latest/writing_plugins.html#static-assets,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",497162288,Plugin documentation should cover how to bundle static/templates in setup.py, https://github.com/simonw/datasette/issues/1042#issuecomment-715643763,https://api.github.com/repos/simonw/datasette/issues/1042,715643763,MDEyOklzc3VlQ29tbWVudDcxNTY0Mzc2Mw==,9599,simonw,2020-10-24T00:34:31Z,2020-10-24T00:34:52Z,OWNER,I'm going to rename that to template variable from `select_templates` to `templates_considered` too.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",727802081,Plugin hook for loading templates, https://github.com/simonw/datasette/issues/1042#issuecomment-715643646,https://api.github.com/repos/simonw/datasette/issues/1042,715643646,MDEyOklzc3VlQ29tbWVudDcxNTY0MzY0Ng==,9599,simonw,2020-10-24T00:33:46Z,2020-10-24T00:33:46Z,OWNER,"I'd like to do this all in the `datasette.render_template()` method to ensure it's available to plugins as well, not just core code that uses the `BaseView` class. This code is the problem: https://github.com/simonw/datasette/blob/d3e9b0aecb6f8e9b2befd9c654ccb7ce852db3e7/datasette/views/base.py#L114-L133 I think I'll fix this by moving the `select_templates` mechanism into `datasette.render_templates()`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",727802081,Plugin hook for loading templates, https://github.com/simonw/datasette/issues/1045#issuecomment-715641183,https://api.github.com/repos/simonw/datasette/issues/1045,715641183,MDEyOklzc3VlQ29tbWVudDcxNTY0MTE4Mw==,9599,simonw,2020-10-24T00:19:29Z,2020-10-24T00:19:29Z,OWNER,"It turns out it already does that: https://github.com/simonw/datasette/blob/976e5f74aae1fa0d406df6691dc8b5feeebe8788/datasette/app.py#L710-L720 But the documentation doesn't reflect that: > `template` - string > > > The template file to be rendered, e.g. `my_plugin.html`. Datasette will search for this file first in the `--template-dir=` location, if it was specified - then in the plugin's bundled templates and finally in Datasette's set of default templates.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",728600048,"Document that datasette.render_template(template, ...) also accepts a list of templates", https://github.com/simonw/datasette/issues/1042#issuecomment-715618333,https://api.github.com/repos/simonw/datasette/issues/1042,715618333,MDEyOklzc3VlQ29tbWVudDcxNTYxODMzMw==,9599,simonw,2020-10-23T22:33:24Z,2020-10-23T22:33:24Z,OWNER,"It wouldn't be a disaster if template-loading plugins were unable to hook into the `/{slug1}/{slug2}.html` custom page mechanism, since plugins can define their own pages already using `register_routes()`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",727802081,Plugin hook for loading templates, https://github.com/simonw/datasette/issues/1042#issuecomment-715618077,https://api.github.com/repos/simonw/datasette/issues/1042,715618077,MDEyOklzc3VlQ29tbWVudDcxNTYxODA3Nw==,9599,simonw,2020-10-23T22:32:24Z,2020-10-23T22:32:24Z,OWNER,"Another option: the first version of the plugin hook could accept only the template filename. Subsequent releases could add more arguments, since Pluggy allows new arguments to be added without breaking backwards compatibility.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",727802081,Plugin hook for loading templates, https://github.com/simonw/datasette/issues/1042#issuecomment-715617830,https://api.github.com/repos/simonw/datasette/issues/1042,715617830,MDEyOklzc3VlQ29tbWVudDcxNTYxNzgzMA==,9599,simonw,2020-10-23T22:31:26Z,2020-10-23T22:31:26Z,OWNER,"So maybe this should be a `register_template_loader` mechanism that returns a Jinja loader after all? That would mean that only the template filename could be used as the input to the plugin, which doesn't seem as useful as emulating the `extra_template_vars()` interface.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",727802081,Plugin hook for loading templates, https://github.com/simonw/datasette/issues/1042#issuecomment-715617405,https://api.github.com/repos/simonw/datasette/issues/1042,715617405,MDEyOklzc3VlQ29tbWVudDcxNTYxNzQwNQ==,9599,simonw,2020-10-23T22:29:53Z,2020-10-23T22:29:53Z,OWNER,"Also consider that `DatasetteRouter` uses `.list_templates()` to gather together `{slug}.html` style templates for the custom page templates mechanism: https://github.com/simonw/datasette/blob/976e5f74aae1fa0d406df6691dc8b5feeebe8788/datasette/app.py#L949-L967 For that to work with the new plugin hook, custom template providing plugins will need a way to provide a list of templates that they know about.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",727802081,Plugin hook for loading templates, https://github.com/simonw/datasette/issues/1042#issuecomment-715616757,https://api.github.com/repos/simonw/datasette/issues/1042,715616757,MDEyOklzc3VlQ29tbWVudDcxNTYxNjc1Nw==,9599,simonw,2020-10-23T22:27:28Z,2020-10-23T22:27:28Z,OWNER,"Almost all of the core template loading happens in the `BaseView.render` method: https://github.com/simonw/datasette/blob/976e5f74aae1fa0d406df6691dc8b5feeebe8788/datasette/views/base.py#L114-L133 The one exception is the 404 handling code here: https://github.com/simonw/datasette/blob/976e5f74aae1fa0d406df6691dc8b5feeebe8788/datasette/app.py#L1034-L1042","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",727802081,Plugin hook for loading templates, https://github.com/simonw/datasette/issues/1042#issuecomment-715614971,https://api.github.com/repos/simonw/datasette/issues/1042,715614971,MDEyOklzc3VlQ29tbWVudDcxNTYxNDk3MQ==,9599,simonw,2020-10-23T22:20:14Z,2020-10-23T22:23:51Z,OWNER,"Alternative plugin hook idea: ```python @hookspec def load_template(template, database, table, columns, view_name, request, datasette): ""Load the specified template, returning the template code as a string"" ``` Imitating the existing `extra_template_vars` family of hooks: https://docs.datasette.io/en/stable/plugin_hooks.html#extra-template-vars-template-database-table-columns-view-name-request-datasette","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",727802081,Plugin hook for loading templates, https://github.com/simonw/datasette/pull/1040#issuecomment-715587715,https://api.github.com/repos/simonw/datasette/issues/1040,715587715,MDEyOklzc3VlQ29tbWVudDcxNTU4NzcxNQ==,9599,simonw,2020-10-23T21:01:07Z,2020-10-23T21:03:10Z,OWNER,"A download icon would be nice for the links in the table display. I like this one https://primer.style/octicons/download-24","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",726910999,/db/table/-/blob/pk/column.blob download URL, https://github.com/simonw/datasette/pull/1043#issuecomment-715586711,https://api.github.com/repos/simonw/datasette/issues/1043,715586711,MDEyOklzc3VlQ29tbWVudDcxNTU4NjcxMQ==,9599,simonw,2020-10-23T20:58:26Z,2020-10-23T20:58:26Z,OWNER,I misunderstood - `asgi-csrf` already has an sdist.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",727915394,Include LICENSE in sdist, https://github.com/simonw/datasette/pull/1043#issuecomment-715585140,https://api.github.com/repos/simonw/datasette/issues/1043,715585140,MDEyOklzc3VlQ29tbWVudDcxNTU4NTE0MA==,9599,simonw,2020-10-23T20:54:29Z,2020-10-23T20:54:29Z,OWNER,Thanks. I'll push a source release of `asgi-csrf`.,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",727915394,Include LICENSE in sdist, https://github.com/simonw/datasette/pull/1044#issuecomment-715584579,https://api.github.com/repos/simonw/datasette/issues/1044,715584579,MDEyOklzc3VlQ29tbWVudDcxNTU4NDU3OQ==,9599,simonw,2020-10-23T20:53:01Z,2020-10-23T20:53:01Z,OWNER,Thanks for this!,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",727916744,Add minimum supported python, https://github.com/simonw/datasette/issues/745#issuecomment-715556545,https://api.github.com/repos/simonw/datasette/issues/745,715556545,MDEyOklzc3VlQ29tbWVudDcxNTU1NjU0NQ==,9599,simonw,2020-10-23T19:47:10Z,2020-10-23T19:47:10Z,OWNER,Dupe of #647 ,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",608613033,Extract the hash-URL mechanism out into a plugin, https://github.com/simonw/datasette/issues/1042#issuecomment-715497419,https://api.github.com/repos/simonw/datasette/issues/1042,715497419,MDEyOklzc3VlQ29tbWVudDcxNTQ5NzQxOQ==,9599,simonw,2020-10-23T18:12:40Z,2020-10-23T18:12:40Z,OWNER,Maybe the template loader can optionally return some extra context to pass to the template. That could be used to solve the templates considered comment.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",727802081,Plugin hook for loading templates, https://github.com/simonw/datasette/issues/1042#issuecomment-715496859,https://api.github.com/repos/simonw/datasette/issues/1042,715496859,MDEyOklzc3VlQ29tbWVudDcxNTQ5Njg1OQ==,9599,simonw,2020-10-23T18:11:27Z,2020-10-23T18:11:27Z,OWNER,"When loading a template the filename is required, but you can optionally also send a set of extra arguments which the template loader can take into consideration.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",727802081,Plugin hook for loading templates, https://github.com/simonw/datasette/issues/1042#issuecomment-715490532,https://api.github.com/repos/simonw/datasette/issues/1042,715490532,MDEyOklzc3VlQ29tbWVudDcxNTQ5MDUzMg==,9599,simonw,2020-10-23T17:57:34Z,2020-10-23T17:57:34Z,OWNER,"A better version of this hook would be passed the database, table and query name depending on what was being rendered. This would require some re-thinking of how core templates are loaded, especially since I would want the templates considered comment to continue working.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",727802081,Plugin hook for loading templates, https://github.com/simonw/datasette/pull/1044#issuecomment-714916127,https://api.github.com/repos/simonw/datasette/issues/1044,714916127,MDEyOklzc3VlQ29tbWVudDcxNDkxNjEyNw==,22429695,codecov[bot],2020-10-23T05:12:52Z,2020-10-23T05:12:52Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1044?src=pr&el=h1) Report > Merging [#1044](https://codecov.io/gh/simonw/datasette/pull/1044?src=pr&el=desc) into [main](https://codecov.io/gh/simonw/datasette/commit/d0cc6f4c32e1f89238ddec782086b3122f445bd4?el=desc) will **not change** coverage. > The diff coverage is `n/a`. [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1044/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1)](https://codecov.io/gh/simonw/datasette/pull/1044?src=pr&el=tree) ```diff @@ Coverage Diff @@ ## main #1044 +/- ## ======================================= Coverage 84.65% 84.65% ======================================= Files 28 28 Lines 3924 3924 ======================================= Hits 3322 3322 Misses 602 602 ``` ------ [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1044?src=pr&el=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1044?src=pr&el=footer). Last update [d0cc6f4...6453ab1](https://codecov.io/gh/simonw/datasette/pull/1044?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",727916744,Add minimum supported python, https://github.com/simonw/datasette/pull/1043#issuecomment-714915025,https://api.github.com/repos/simonw/datasette/issues/1043,714915025,MDEyOklzc3VlQ29tbWVudDcxNDkxNTAyNQ==,22429695,codecov[bot],2020-10-23T05:09:09Z,2020-10-23T05:09:09Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1043?src=pr&el=h1) Report > Merging [#1043](https://codecov.io/gh/simonw/datasette/pull/1043?src=pr&el=desc) into [main](https://codecov.io/gh/simonw/datasette/commit/d0cc6f4c32e1f89238ddec782086b3122f445bd4?el=desc) will **not change** coverage. > The diff coverage is `n/a`. [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1043/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1)](https://codecov.io/gh/simonw/datasette/pull/1043?src=pr&el=tree) ```diff @@ Coverage Diff @@ ## main #1043 +/- ## ======================================= Coverage 84.65% 84.65% ======================================= Files 28 28 Lines 3924 3924 ======================================= Hits 3322 3322 Misses 602 602 ``` ------ [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1043?src=pr&el=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1043?src=pr&el=footer). Last update [d0cc6f4...dc4129c](https://codecov.io/gh/simonw/datasette/pull/1043?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",727915394,Include LICENSE in sdist, https://github.com/simonw/datasette/issues/1012#issuecomment-714908859,https://api.github.com/repos/simonw/datasette/issues/1012,714908859,MDEyOklzc3VlQ29tbWVudDcxNDkwODg1OQ==,45380,bollwyvl,2020-10-23T04:49:20Z,2020-10-23T04:49:20Z,CONTRIBUTOR,"Good luck on 1.0! It may also be worth lobbying for a `Framework::Datasette::1.0` classifier. This would be a nice way to allow the ecosystem to self-document a bit more [discoverably](https://pypi.org/search/?q=&o=&c=Framework+%3A%3A+Datasette%3A%3A+1.0). I was surprised to see the [PR for `Framework::Jupyter`](https://github.com/pypa/warehouse/pull/1905/files) is a... database migration! Of course, there may be more workflow to it!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",718540751,For 1.0 update trove classifier in setup.py, https://github.com/simonw/datasette/issues/1042#issuecomment-714868867,https://api.github.com/repos/simonw/datasette/issues/1042,714868867,MDEyOklzc3VlQ29tbWVudDcxNDg2ODg2Nw==,9599,simonw,2020-10-23T02:31:17Z,2020-10-23T02:31:17Z,OWNER,I'll build this in conjunction with a plugin that supports editing templates stored in SQLite.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",727802081,Plugin hook for loading templates, https://github.com/simonw/datasette/issues/1042#issuecomment-714868624,https://api.github.com/repos/simonw/datasette/issues/1042,714868624,MDEyOklzc3VlQ29tbWVudDcxNDg2ODYyNA==,9599,simonw,2020-10-23T02:30:27Z,2020-10-23T02:30:37Z,OWNER,Maybe `register_template_loader(datasette)` which returns an object which is added in at the beginning of the list passed to `ChoiceLoader` here.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",727802081,Plugin hook for loading templates, https://github.com/simonw/datasette/issues/1042#issuecomment-714868207,https://api.github.com/repos/simonw/datasette/issues/1042,714868207,MDEyOklzc3VlQ29tbWVudDcxNDg2ODIwNw==,9599,simonw,2020-10-23T02:29:12Z,2020-10-23T02:29:12Z,OWNER,Relevant code: https://github.com/simonw/datasette/blob/d0cc6f4c32e1f89238ddec782086b3122f445bd4/datasette/app.py#L288-L311,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",727802081,Plugin hook for loading templates, https://github.com/simonw/sqlite-utils/issues/173#issuecomment-714758139,https://api.github.com/repos/simonw/sqlite-utils/issues/173,714758139,MDEyOklzc3VlQ29tbWVudDcxNDc1ODEzOQ==,9599,simonw,2020-10-22T20:57:56Z,2020-10-22T20:57:56Z,OWNER,"I could use `ijson` to provide a progress bar for JSON arrays too. I'd prefer to keep that as an optional dependency though, since `sqlite-utils` is a library dependency for many other projects and it would be using `ijson` purely for the CLI component. Here's how to iterate through a list of objects being read from a file: ```python import json parser = ijson.items(open( ""/tmp/list.json"" ), ""item"") for object in parser: # ... ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",707478649,Progress bar for sqlite-utils insert, https://github.com/simonw/datasette/issues/1041#issuecomment-714683801,https://api.github.com/repos/simonw/datasette/issues/1041,714683801,MDEyOklzc3VlQ29tbWVudDcxNDY4MzgwMQ==,9599,simonw,2020-10-22T18:37:47Z,2020-10-22T18:37:47Z,OWNER,"I think I'll do this by looking for URLs that start with `/` - since it's also possible to have full `https://...` URLs in that setting. ```json { ""extra_css_urls"": [ ""/static/styles.css"" ], ""extra_js_urls"": [ ""/static/app.js"" ] } ``` I need to think about the `extra_css_urls` and `extra_js_urls` plugin hooks too: https://docs.datasette.io/en/stable/plugin_hooks.html#extra-css-urls-template-database-table-columns-view-name-request-datasette","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",727627923,extra_js_urls and extra_css_urls should respect base_url setting, https://github.com/simonw/datasette/issues/1041#issuecomment-714682825,https://api.github.com/repos/simonw/datasette/issues/1041,714682825,MDEyOklzc3VlQ29tbWVudDcxNDY4MjgyNQ==,9599,simonw,2020-10-22T18:36:10Z,2020-10-22T18:36:10Z,OWNER,I'll need to update these docs once there's a solution for this in place: https://docs.datasette.io/en/latest/custom_templates.html#serving-static-files,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",727627923,extra_js_urls and extra_css_urls should respect base_url setting, https://github.com/simonw/datasette/issues/1041#issuecomment-714682288,https://api.github.com/repos/simonw/datasette/issues/1041,714682288,MDEyOklzc3VlQ29tbWVudDcxNDY4MjI4OA==,9599,simonw,2020-10-22T18:35:15Z,2020-10-22T18:35:15Z,OWNER,"@psychemedia said: https://github.com/simonw/datasette/issues/1033#issuecomment-714657366 > How does `/-/static` relate to [current guidance docs around `static`](https://docs.datasette.io/en/latest/custom_templates.html?highlight=static#serving-static-files) regarding the `--static option` and metadata formulations such as `""extra_js_urls"": [ ""/static/app.js""]` (I've not managed to get this to work in a Jupyter server proxied set up; the [datasette / jupyter server proxy repo](https://github.com/simonw/jupyterserverproxy-datasette-demo) may provide a useful test example, eg via MyBinder, for folk to crib from?)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",727627923,extra_js_urls and extra_css_urls should respect base_url setting, https://github.com/simonw/datasette/issues/1033#issuecomment-714681365,https://api.github.com/repos/simonw/datasette/issues/1033,714681365,MDEyOklzc3VlQ29tbWVudDcxNDY4MTM2NQ==,9599,simonw,2020-10-22T18:33:48Z,2020-10-22T18:33:48Z,OWNER,"That's a good question - I hadn't considered that. I'm going to open a new issue to have `extra_js_urls` respect the `base_url` setting, since the static files will be served from a different location.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725099777,datasette.urls.static_plugins(...) method, https://github.com/simonw/datasette/issues/1033#issuecomment-714657366,https://api.github.com/repos/simonw/datasette/issues/1033,714657366,MDEyOklzc3VlQ29tbWVudDcxNDY1NzM2Ng==,82988,psychemedia,2020-10-22T17:51:29Z,2020-10-22T17:51:29Z,CONTRIBUTOR,"How does `/-/static` relate to [current guidance docs around `static`](https://docs.datasette.io/en/latest/custom_templates.html?highlight=static#serving-static-files) regarding the `--static option` and metadata formulations such as `""extra_js_urls"": [ ""/static/app.js""]` (I've not managed to get this to work in a Jupyter server proxied set up; the [datasette / jupyter server proxy repo](https://github.com/simonw/jupyterserverproxy-datasette-demo) may provide a useful test example, eg via MyBinder, for folk to crib from?) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725099777,datasette.urls.static_plugins(...) method, https://github.com/simonw/datasette/pull/1031#issuecomment-714289680,https://api.github.com/repos/simonw/datasette/issues/1031,714289680,MDEyOklzc3VlQ29tbWVudDcxNDI4OTY4MA==,299380,frankier,2020-10-22T07:23:52Z,2020-10-22T07:23:52Z,NONE,"The bug is that currently when there are databases passed in, but no -i flag, e.g. in configuration directory mode, inclusion in inspect-data.json does not automatically cause databases to be considered immutable, as described in the documentation. The reason is that the -i flag is specified multiple=True, which means when it is not passed in we will get an empty list [], rather than None. So the current code decides that no databases are immutable rather than falling back to inspect-data.json -- as is presumably intended.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",724369025,Fallback to databases in inspect-data.json when no -i options are passed, https://github.com/simonw/sqlite-utils/issues/171#issuecomment-714219725,https://api.github.com/repos/simonw/sqlite-utils/issues/171,714219725,MDEyOklzc3VlQ29tbWVudDcxNDIxOTcyNQ==,649467,mhalle,2020-10-22T04:38:35Z,2020-10-22T04:38:35Z,NONE,"Thanks. As I said, I think the result (being able to query tree structures like ancestors and descendants) is more important than the implementation, and I agree that this particular sqlite extension is too obscure. Just providing an sqlite utility to build or rebuild a transitive closure table might be more generically useful. I find that hierarchical data shows up pretty frequently in some data science problems.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",707407567,Idea: transitive closure tables for tree structures, https://github.com/simonw/sqlite-utils/issues/171#issuecomment-714208848,https://api.github.com/repos/simonw/sqlite-utils/issues/171,714208848,MDEyOklzc3VlQ29tbWVudDcxNDIwODg0OA==,9599,simonw,2020-10-22T04:07:14Z,2020-10-22T04:07:14Z,OWNER,"I made the `--load-extension` command much more widely supported in #137 - which should be useful for anyone who wants to use this extension. It's a bit too obscure for me to want to add direct Python library support relating to that extension though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",707407567,Idea: transitive closure tables for tree structures, https://github.com/simonw/datasette/pull/1031#issuecomment-714206875,https://api.github.com/repos/simonw/datasette/issues/1031,714206875,MDEyOklzc3VlQ29tbWVudDcxNDIwNjg3NQ==,9599,simonw,2020-10-22T04:01:19Z,2020-10-22T04:01:19Z,OWNER,I don't fully understand the bug you're fixing here. Could you provide a bit more explanation?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",724369025,Fallback to databases in inspect-data.json when no -i options are passed, https://github.com/simonw/datasette/pull/1040#issuecomment-714206533,https://api.github.com/repos/simonw/datasette/issues/1040,714206533,MDEyOklzc3VlQ29tbWVudDcxNDIwNjUzMw==,9599,simonw,2020-10-22T04:00:25Z,2020-10-22T04:00:25Z,OWNER,I've decided not to offer a configuration option to turn this off. I'll reconsider if someone asks for it.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",726910999,/db/table/-/blob/pk/column.blob download URL, https://github.com/simonw/datasette/issues/998#issuecomment-714205783,https://api.github.com/repos/simonw/datasette/issues/998,714205783,MDEyOklzc3VlQ29tbWVudDcxNDIwNTc4Mw==,9599,simonw,2020-10-22T03:58:13Z,2020-10-22T03:58:13Z,OWNER,This is now live here: https://global-power-plants.datasettes.com/global-power-plants/global-power-plants,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",717699884,Wide tables should scroll horizontally within the page, https://github.com/simonw/datasette/issues/998#issuecomment-714117534,https://api.github.com/repos/simonw/datasette/issues/998,714117534,MDEyOklzc3VlQ29tbWVudDcxNDExNzUzNA==,9599,simonw,2020-10-22T01:12:06Z,2020-10-22T01:12:06Z,OWNER,"Demo: ![table-scroll](https://user-images.githubusercontent.com/9599/96806421-e74e7e00-13c8-11eb-95fe-44d01e4c2eb3.gif) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",717699884,Wide tables should scroll horizontally within the page, https://github.com/simonw/datasette/issues/998#issuecomment-714092002,https://api.github.com/repos/simonw/datasette/issues/998,714092002,MDEyOklzc3VlQ29tbWVudDcxNDA5MjAwMg==,9599,simonw,2020-10-22T00:55:10Z,2020-10-22T00:55:10Z,OWNER,This isn't blocked on #987 - it just means that `datasette-cluster-map` will need to learn to look for `.table-wrapper` first and fall back on the table.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",717699884,Wide tables should scroll horizontally within the page, https://github.com/simonw/datasette/issues/998#issuecomment-714090965,https://api.github.com/repos/simonw/datasette/issues/998,714090965,MDEyOklzc3VlQ29tbWVudDcxNDA5MDk2NQ==,9599,simonw,2020-10-22T00:54:30Z,2020-10-22T00:54:30Z,OWNER,"Easiest fix for the column action menu positioning - hide them when the user scrolls the containing div: ```javascript document.querySelector('.table-wrapper').addEventListener( 'scroll', () => document.querySelector('.dropdown-menu').style.display = 'none' ); ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",717699884,Wide tables should scroll horizontally within the page, https://github.com/simonw/datasette/pull/1038#issuecomment-713920461,https://api.github.com/repos/simonw/datasette/issues/1038,713920461,MDEyOklzc3VlQ29tbWVudDcxMzkyMDQ2MQ==,9599,simonw,2020-10-21T22:43:51Z,2020-10-21T22:43:51Z,OWNER,Thanks for spotting this!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",726154220,DOC: Fix syntax error, https://github.com/simonw/datasette/issues/1036#issuecomment-713899530,https://api.github.com/repos/simonw/datasette/issues/1036,713899530,MDEyOklzc3VlQ29tbWVudDcxMzg5OTUzMA==,9599,simonw,2020-10-21T21:55:00Z,2020-10-21T21:55:00Z,OWNER,"This code needs these permission checks: https://github.com/simonw/datasette/blob/bf82b3d6a605c9ddadd5fb739249dfe6defaf635/datasette/views/table.py#L911-L913","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725996507,Make it possible to download BLOB data from the Datasette UI, https://github.com/simonw/datasette/issues/1036#issuecomment-713821656,https://api.github.com/repos/simonw/datasette/issues/1036,713821656,MDEyOklzc3VlQ29tbWVudDcxMzgyMTY1Ng==,9599,simonw,2020-10-21T19:22:45Z,2020-10-21T19:41:48Z,OWNER,"So for https://latest.datasette.io/fixtures/binary_data the BLOB download URLs would be: `https://latest.datasette.io/fixtures/-/blob/binary_data/1/data.blob` - that last bit after the primary key is to indicate the `data` column With these headers: - `Content-Disposition: attachment; filename=""binary_data-1-data.blob""` - `X-Content-Type-Options: nosniff` - `Content-Type: application/binary`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725996507,Make it possible to download BLOB data from the Datasette UI, https://github.com/simonw/datasette/issues/1036#issuecomment-713830842,https://api.github.com/repos/simonw/datasette/issues/1036,713830842,MDEyOklzc3VlQ29tbWVudDcxMzgzMDg0Mg==,9599,simonw,2020-10-21T19:41:20Z,2020-10-21T19:41:20Z,OWNER,Another useful demo database: https://datasette-render-images-demo.datasette.io/favicons/favicons - see https://datasette-render-images-demo.datasette.io/favicons/favicons.csv,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725996507,Make it possible to download BLOB data from the Datasette UI, https://github.com/simonw/datasette/issues/1036#issuecomment-713829629,https://api.github.com/repos/simonw/datasette/issues/1036,713829629,MDEyOklzc3VlQ29tbWVudDcxMzgyOTYyOQ==,9599,simonw,2020-10-21T19:38:43Z,2020-10-21T19:38:43Z,OWNER,"Should this work just for BLOB columns, or should it work for other columns too? For the moment I'm going to restrict it to BLOBs, since data from other columns is available through the UI whereas BLOB columns are not.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725996507,Make it possible to download BLOB data from the Datasette UI, https://github.com/simonw/datasette/issues/1036#issuecomment-713818817,https://api.github.com/repos/simonw/datasette/issues/1036,713818817,MDEyOklzc3VlQ29tbWVudDcxMzgxODgxNw==,9599,simonw,2020-10-21T19:17:01Z,2020-10-21T19:17:01Z,OWNER,Actually I like `.blob`,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725996507,Make it possible to download BLOB data from the Datasette UI, https://github.com/simonw/datasette/issues/1036#issuecomment-713818178,https://api.github.com/repos/simonw/datasette/issues/1036,713818178,MDEyOklzc3VlQ29tbWVudDcxMzgxODE3OA==,9599,simonw,2020-10-21T19:15:38Z,2020-10-21T19:16:34Z,OWNER,"What should the suggested filename be? I think something that includes the table name, primary key and the name of the column would work. How about a file extension? I guess `.binary`, then let the user rename it? Or `.raw`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725996507,Make it possible to download BLOB data from the Datasette UI, https://github.com/simonw/datasette/issues/1039#issuecomment-713754844,https://api.github.com/repos/simonw/datasette/issues/1039,713754844,MDEyOklzc3VlQ29tbWVudDcxMzc1NDg0NA==,9599,simonw,2020-10-21T17:58:27Z,2020-10-21T17:58:27Z,OWNER,"Now live: https://latest.datasette.io/fixtures/roadside_attraction_characteristics ![anim](https://user-images.githubusercontent.com/9599/96759016-55288480-138c-11eb-8ba0-d8e0f6dd8b1f.gif) ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",726687572,Add an animation to the column actions menu, https://github.com/simonw/datasette/pull/1038#issuecomment-713320666,https://api.github.com/repos/simonw/datasette/issues/1038,713320666,MDEyOklzc3VlQ29tbWVudDcxMzMyMDY2Ng==,22429695,codecov[bot],2020-10-21T05:50:38Z,2020-10-21T05:50:38Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1038?src=pr&el=h1) Report > Merging [#1038](https://codecov.io/gh/simonw/datasette/pull/1038?src=pr&el=desc) into [main](https://codecov.io/gh/simonw/datasette/commit/66120a7a1cb592e8a21164cf537f62a4d7ab1dfc?el=desc) will **not change** coverage. > The diff coverage is `n/a`. [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1038/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1)](https://codecov.io/gh/simonw/datasette/pull/1038?src=pr&el=tree) ```diff @@ Coverage Diff @@ ## main #1038 +/- ## ======================================= Coverage 84.65% 84.65% ======================================= Files 28 28 Lines 3924 3924 ======================================= Hits 3322 3322 Misses 602 602 ``` ------ [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1038?src=pr&el=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1038?src=pr&el=footer). Last update [66120a7...7fc0cce](https://codecov.io/gh/simonw/datasette/pull/1038?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",726154220,DOC: Fix syntax error, https://github.com/simonw/datasette/issues/1036#issuecomment-713278349,https://api.github.com/repos/simonw/datasette/issues/1036,713278349,MDEyOklzc3VlQ29tbWVudDcxMzI3ODM0OQ==,9599,simonw,2020-10-21T03:42:29Z,2020-10-21T03:42:29Z,OWNER,Possible URL for this: `/db/table/-/blob/primary-keys` - this would use the `/db/table/-/` namespace proposed in #296.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725996507,Make it possible to download BLOB data from the Datasette UI, https://github.com/simonw/datasette/issues/998#issuecomment-713269155,https://api.github.com/repos/simonw/datasette/issues/998,713269155,MDEyOklzc3VlQ29tbWVudDcxMzI2OTE1NQ==,9599,simonw,2020-10-21T03:17:07Z,2020-10-21T03:17:07Z,OWNER,"This may require updates to the column action menu JavaScript too, since it was not built with scrolling sideways in mind.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",717699884,Wide tables should scroll horizontally within the page, https://github.com/simonw/datasette/issues/1037#issuecomment-713268905,https://api.github.com/repos/simonw/datasette/issues/1037,713268905,MDEyOklzc3VlQ29tbWVudDcxMzI2ODkwNQ==,9599,simonw,2020-10-21T03:16:36Z,2020-10-21T03:16:36Z,OWNER,Dupe of #998.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",726094754,Add horizontal scrollbar to tables, https://github.com/simonw/datasette/issues/1037#issuecomment-713268498,https://api.github.com/repos/simonw/datasette/issues/1037,713268498,MDEyOklzc3VlQ29tbWVudDcxMzI2ODQ5OA==,9599,simonw,2020-10-21T03:15:44Z,2020-10-21T03:15:44Z,OWNER,"This may require updates to the column action menu JavaScript too, since it was not built with scrolling sideways in mind.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",726094754,Add horizontal scrollbar to tables, https://github.com/simonw/datasette/issues/1037#issuecomment-713267989,https://api.github.com/repos/simonw/datasette/issues/1037,713267989,MDEyOklzc3VlQ29tbWVudDcxMzI2Nzk4OQ==,9599,simonw,2020-10-21T03:14:34Z,2020-10-21T03:14:34Z,OWNER,"This is particularly relevant to the `datasette-cluster-map` plugin - the map is much nicer to use if the table itself can be scrolled. That plugin also makes this harder to build, because the plugin inserts the map as the direct predecessor of the `` element and hence breaks if you try to wrap that in a `
`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",726094754,Add horizontal scrollbar to tables, https://github.com/simonw/datasette/issues/1036#issuecomment-713226726,https://api.github.com/repos/simonw/datasette/issues/1036,713226726,MDEyOklzc3VlQ29tbWVudDcxMzIyNjcyNg==,9599,simonw,2020-10-21T01:04:25Z,2020-10-21T01:04:25Z,OWNER,"Extra security idea: a `blob_download_host` setting which can be used to indicate a host that should be used for downloads - for example `datasettestatic.com`. If this setting is populated then binary downloads are served from paths on that host only, and no other Datasette URLs from that host will be served.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725996507,Make it possible to download BLOB data from the Datasette UI, https://github.com/simonw/datasette/issues/262#issuecomment-713208667,https://api.github.com/repos/simonw/datasette/issues/262,713208667,MDEyOklzc3VlQ29tbWVudDcxMzIwODY2Nw==,9599,simonw,2020-10-21T00:03:18Z,2020-10-21T00:03:18Z,OWNER,"I think I should prioritize the facets component of this, since that could have significant performance wins while also supporting `datasette-graphql`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",323658641,Add ?_extra= mechanism for requesting extra properties in JSON, https://github.com/simonw/datasette/issues/262#issuecomment-713200782,https://api.github.com/repos/simonw/datasette/issues/262,713200782,MDEyOklzc3VlQ29tbWVudDcxMzIwMDc4Mg==,9599,simonw,2020-10-20T23:41:30Z,2020-10-20T23:41:30Z,OWNER,This is now blocking https://github.com/simonw/datasette-graphql/issues/61 because that issue needs a way to turn off suggested facets when retrieving the results of a table query.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",323658641,Add ?_extra= mechanism for requesting extra properties in JSON, https://github.com/simonw/datasette/issues/1034#issuecomment-713191819,https://api.github.com/repos/simonw/datasette/issues/1034,713191819,MDEyOklzc3VlQ29tbWVudDcxMzE5MTgxOQ==,9599,simonw,2020-10-20T23:12:58Z,2020-10-20T23:12:58Z,OWNER,"Enzo has a great solution here: https://twitter.com/enzo_mdd/status/1318685442976436226 > Or maybe an option for a url. This keeps the CSV small but allows scripts to download binary data as needed. In #1036 I'm planning on adding a way for users to access BLOB data. I can include that URL in the CSV output.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725184645,Better way of representing binary data in .csv output, https://github.com/simonw/datasette/issues/1036#issuecomment-713186189,https://api.github.com/repos/simonw/datasette/issues/1036,713186189,MDEyOklzc3VlQ29tbWVudDcxMzE4NjE4OQ==,9599,simonw,2020-10-20T22:56:33Z,2020-10-20T22:56:33Z,OWNER,I think this plus the binary-CSV stuff in #1034 will justify a dedicated section of the documentation to talk about how Datasette handles binary BLOB columns.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725996507,Make it possible to download BLOB data from the Datasette UI, https://github.com/simonw/datasette/issues/1036#issuecomment-713185871,https://api.github.com/repos/simonw/datasette/issues/1036,713185871,MDEyOklzc3VlQ29tbWVudDcxMzE4NTg3MQ==,9599,simonw,2020-10-20T22:55:36Z,2020-10-20T22:55:36Z,OWNER,I can also use a `Content-Disposition` header to force a download. I'm reasonably confident that the combination of `Content-Disposition` and `X-Content-Type-Options: nosniff` and `application/binary` will let me allow users to download the contents of arbitrary BLOB columns without any XSS risk.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725996507,Make it possible to download BLOB data from the Datasette UI, https://github.com/simonw/datasette/issues/1036#issuecomment-713185173,https://api.github.com/repos/simonw/datasette/issues/1036,713185173,MDEyOklzc3VlQ29tbWVudDcxMzE4NTE3Mw==,9599,simonw,2020-10-20T22:53:41Z,2020-10-20T22:53:41Z,OWNER,"https://security.stackexchange.com/questions/12896/does-x-content-type-options-really-prevent-content-sniffing-attacks says: > In Tangled Web Michal Zalewski says: > > > Refrain from using Content-Type: application/octet-stream and use application/binary instead, especially for unknown document types. Refrain from returning Content-Type: text/plain. > > > > For example, any code-hosting platform must exercise caution when returning executables or source archives as application/octet-stream, because there is a risk they may be misinterpreted as HTML and displayed inline.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725996507,Make it possible to download BLOB data from the Datasette UI, https://github.com/simonw/datasette/issues/1036#issuecomment-713184374,https://api.github.com/repos/simonw/datasette/issues/1036,713184374,MDEyOklzc3VlQ29tbWVudDcxMzE4NDM3NA==,9599,simonw,2020-10-20T22:51:22Z,2020-10-20T22:51:22Z,OWNER,"From https://hackerone.com/reports/126197: > archive.uber.com mirrors pypi. When downloading `.tar.gz` files from archive.uber.com, the MIME type is `application/octet-stream`. Injecting `` into the start of the `.tar.gz` causes an XSS in Internet Explorer due to MIME sniffing. So you do have to be careful not to open accidental XSS holes with `application/octet-stream` thanks to (presumably older) versions of IE. From that thread it looks like the solution is to add a `X-Content-Type-Options: nosniff` header.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725996507,Make it possible to download BLOB data from the Datasette UI, https://github.com/simonw/datasette/issues/1036#issuecomment-713183306,https://api.github.com/repos/simonw/datasette/issues/1036,713183306,MDEyOklzc3VlQ29tbWVudDcxMzE4MzMwNg==,9599,simonw,2020-10-20T22:48:10Z,2020-10-20T22:48:10Z,OWNER,Twitter thread: https://twitter.com/dancow/status/1318681053347840005,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725996507,Make it possible to download BLOB data from the Datasette UI, https://github.com/simonw/datasette/issues/1034#issuecomment-713176082,https://api.github.com/repos/simonw/datasette/issues/1034,713176082,MDEyOklzc3VlQ29tbWVudDcxMzE3NjA4Mg==,9599,simonw,2020-10-20T22:27:33Z,2020-10-20T22:27:33Z,OWNER,"This feels good to me - it's consistent with how other features in Datasette work, and it means users who need the binary data in CSV (for whatever reason) can get it if they want to.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725184645,Better way of representing binary data in .csv output, https://github.com/simonw/datasette/issues/1034#issuecomment-713175741,https://api.github.com/repos/simonw/datasette/issues/1034,713175741,MDEyOklzc3VlQ29tbWVudDcxMzE3NTc0MQ==,9599,simonw,2020-10-20T22:26:45Z,2020-10-20T22:26:45Z,OWNER,"> New idea: since binary in CSV doesn't make sense anyway, emulate Datasette's HTML UI default and output this: > > id,title,data > 1,Some title, > 2,Other title, > > Then allow users to add ?_base64=1 to the URL to get base64 instead > https://twitter.com/simonw/status/1318679950635888641","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725184645,Better way of representing binary data in .csv output, https://github.com/simonw/datasette/issues/1034#issuecomment-713174690,https://api.github.com/repos/simonw/datasette/issues/1034,713174690,MDEyOklzc3VlQ29tbWVudDcxMzE3NDY5MA==,9599,simonw,2020-10-20T22:23:50Z,2020-10-20T22:23:50Z,OWNER,Or... default to `` and support a `?_base64=1` option which outputs in base64 instead.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725184645,Better way of representing binary data in .csv output, https://github.com/simonw/datasette/issues/1034#issuecomment-713174341,https://api.github.com/repos/simonw/datasette/issues/1034,713174341,MDEyOklzc3VlQ29tbWVudDcxMzE3NDM0MQ==,9599,simonw,2020-10-20T22:22:53Z,2020-10-20T22:23:14Z,OWNER,"An even easier option: do what the Datasette UI does and output `` for that CSV cell, as seen on https://latest.datasette.io/fixtures/binary_data","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725184645,Better way of representing binary data in .csv output, https://github.com/simonw/datasette/issues/1034#issuecomment-713172901,https://api.github.com/repos/simonw/datasette/issues/1034,713172901,MDEyOklzc3VlQ29tbWVudDcxMzE3MjkwMQ==,9599,simonw,2020-10-20T22:19:10Z,2020-10-20T22:20:28Z,OWNER,"I could go with the same format as `datasette-render-binary` but using `0x00` as the format for the hex bytes. 0x15 0x1C 0x02 0xC7 JFIF 0x00 0x01 Problem with this is that it's ambiguous: if the ASCII characters `0x15` occur in the text they will be indistinguishable from those hex bytes. But since representing binary data in CSV fundamentally doesn't make sense I'm not sure if that really matters.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725184645,Better way of representing binary data in .csv output, https://github.com/simonw/datasette/issues/741#issuecomment-713171742,https://api.github.com/repos/simonw/datasette/issues/741,713171742,MDEyOklzc3VlQ29tbWVudDcxMzE3MTc0Mg==,9599,simonw,2020-10-20T22:16:25Z,2020-10-20T22:16:25Z,OWNER,See also #992 which will rename `--config` to `--setting`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",607223136,"Replace ""datasette publish --extra-options"" with ""--setting""", https://github.com/simonw/datasette/issues/262#issuecomment-713170979,https://api.github.com/repos/simonw/datasette/issues/262,713170979,MDEyOklzc3VlQ29tbWVudDcxMzE3MDk3OQ==,9599,simonw,2020-10-20T22:14:37Z,2020-10-20T22:14:37Z,OWNER,"I think it's worth having a plugin hook for this - it can be same hook that is used internally. Maybe `register_extra` - it lets you return one or more `extra` implementations, each with a name and an async function that gets called. Things like suggested facets will become `register_extra` hooks. Maybe actual facets too?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",323658641,Add ?_extra= mechanism for requesting extra properties in JSON, https://github.com/simonw/datasette/issues/262#issuecomment-713170284,https://api.github.com/repos/simonw/datasette/issues/262,713170284,MDEyOklzc3VlQ29tbWVudDcxMzE3MDI4NA==,9599,simonw,2020-10-20T22:13:01Z,2020-10-20T22:13:01Z,OWNER,In the documentation for `?_extra=` I think I'll emphasize the comma-separated version of it. Also: there will be `?_extra=` values which act as aliases for collection combinations - e.g. `?_extra=full` will toggle everything.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",323658641,Add ?_extra= mechanism for requesting extra properties in JSON, https://github.com/simonw/datasette/issues/782#issuecomment-712986115,https://api.github.com/repos/simonw/datasette/issues/782,712986115,MDEyOklzc3VlQ29tbWVudDcxMjk4NjExNQ==,9599,simonw,2020-10-20T16:28:46Z,2020-10-20T16:29:51Z,OWNER,"I think this all comes down to how the `?_extras=` mechanism works (see #262), as first hinted at in a30c5b220c15360d575e94b0e67f3255e120b916 (see commit message) when I added this long-forgotten undocumented feature: https://latest.datasette.io/fixtures/attraction_characteristic/2.json?_extras=foreign_key_tables Extras need to be able to execute additional SQL, since that would solve the problem we have now where the expensive ""suggested facets"" code runs on all `.json` output even when its results are not being shown.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format, https://github.com/simonw/datasette/issues/1035#issuecomment-712976314,https://api.github.com/repos/simonw/datasette/issues/1035,712976314,MDEyOklzc3VlQ29tbWVudDcxMjk3NjMxNA==,9599,simonw,2020-10-20T16:21:42Z,2020-10-20T16:21:42Z,OWNER,"Makes me question if `datasette.urls` should grow functionality equivalent to the other path and querystring manipulation methods in `datasette.utils`: https://github.com/simonw/datasette/blob/66120a7a1cb592e8a21164cf537f62a4d7ab1dfc/datasette/utils/__init__.py#L216-L279","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725743755,"datasette.urls.table(..., format=""json"") argument", https://github.com/simonw/datasette/issues/1035#issuecomment-712965574,https://api.github.com/repos/simonw/datasette/issues/1035,712965574,MDEyOklzc3VlQ29tbWVudDcxMjk2NTU3NA==,9599,simonw,2020-10-20T16:13:57Z,2020-10-20T16:13:57Z,OWNER,"That `renderers[key] = path_with_format(` is in a base class which can be used for both arbitrary queries, canned queries and the table view. I think that's OK, but it means that the `format=""json""` argument on `datasette.urls.table()` won't be used by Datasette internally, it will just be available for plugins.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725743755,"datasette.urls.table(..., format=""json"") argument", https://github.com/simonw/datasette/issues/1035#issuecomment-712963959,https://api.github.com/repos/simonw/datasette/issues/1035,712963959,MDEyOklzc3VlQ29tbWVudDcxMjk2Mzk1OQ==,9599,simonw,2020-10-20T16:11:25Z,2020-10-20T16:11:25Z,OWNER,"Relevant code: https://github.com/simonw/datasette/blob/091441a4449beae559a8c0d007376dc85d3aa624/datasette/utils/__init__.py#L681-L696 Only used here: https://github.com/simonw/datasette/blob/091441a4449beae559a8c0d007376dc85d3aa624/datasette/views/base.py#L498-L502","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725743755,"datasette.urls.table(..., format=""json"") argument", https://github.com/simonw/datasette/issues/1026#issuecomment-712962517,https://api.github.com/repos/simonw/datasette/issues/1026,712962517,MDEyOklzc3VlQ29tbWVudDcxMjk2MjUxNw==,9599,simonw,2020-10-20T16:09:12Z,2020-10-20T16:09:12Z,OWNER,"That `datasette.urls.table(""db"", ""table"") + "".json""` example is bad because if the table name contains a `.` it should be `?_format=json` instead. Maybe `.table()` should have a `format=""json""` option that knows how to do this.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722738988,How should datasette.client interact with base_url, https://github.com/simonw/datasette/issues/1026#issuecomment-712959034,https://api.github.com/repos/simonw/datasette/issues/1026,712959034,MDEyOklzc3VlQ29tbWVudDcxMjk1OTAzNA==,9599,simonw,2020-10-20T16:03:33Z,2020-10-20T16:03:55Z,OWNER,"Reconsidering this: I think the `.get()` etc methods should automatically add the `base_url` prefix for you, since these APIs are only intended to make internal calls. The clincher on this is when I went to add a section to the `datasette.client` documentation recommending you use `datasette.urls.path()` for every call to them that you make. But there's a problem: to handle table name escaping users are likely to want to use `datasette.urls.table()` anyway, like this: response = await datasette.client.get(datasette.urls.table(""db"", ""table"") + "".json"") This risks adding the `base_url` prefix twice. Maybe the `.table()` method could return a string-like object that is marked as already having the `base_url` prefix added, so the `client.get()` method knows not to add it again. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722738988,How should datasette.client interact with base_url, https://github.com/simonw/datasette/issues/991#issuecomment-712855389,https://api.github.com/repos/simonw/datasette/issues/991,712855389,MDEyOklzc3VlQ29tbWVudDcxMjg1NTM4OQ==,24740,furilo,2020-10-20T13:36:41Z,2020-10-20T13:36:41Z,NONE,"Here is one quick sketch (done in Figma :P) for an idea: a possible filter to switch between showing all tables from all databases, or grouping tables by database. (the switch is interactive) All tables: https://www.figma.com/proto/BjFrMroEtmVx6EeRjvSrox/Datasette-test?node-id=1%3A2&viewport=536%2C348%2C0.5&scaling=min-zoom Grouped: https://www.figma.com/proto/BjFrMroEtmVx6EeRjvSrox/Datasette-test?node-id=3%3A974&viewport=536%2C348%2C0.5&scaling=min-zoom When only 1 database: https://www.figma.com/proto/BjFrMroEtmVx6EeRjvSrox/Datasette-test?node-id=1%3A162&viewport=536%2C348%2C0.5&scaling=min-zoom Is this is useful, I can send some more suggestions/sketches. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",714377268,Redesign application homepage, https://github.com/simonw/datasette/issues/1023#issuecomment-712607608,https://api.github.com/repos/simonw/datasette/issues/1023,712607608,MDEyOklzc3VlQ29tbWVudDcxMjYwNzYwOA==,9599,simonw,2020-10-20T05:47:42Z,2020-10-20T05:47:42Z,OWNER,Requested alpha testers in https://github.com/simonw/datasette/issues/838#issuecomment-712604364,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722673818,Fix issues relating to base_url, https://github.com/simonw/datasette/issues/1026#issuecomment-712607227,https://api.github.com/repos/simonw/datasette/issues/1026,712607227,MDEyOklzc3VlQ29tbWVudDcxMjYwNzIyNw==,9599,simonw,2020-10-20T05:46:44Z,2020-10-20T05:46:44Z,OWNER,We have a solution for this now: `datasette.urls` from #1033 can be used by plugins to assemble the correct URLs to pass to `.get()` and friends.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722738988,How should datasette.client interact with base_url, https://github.com/simonw/datasette/issues/1023#issuecomment-712604541,https://api.github.com/repos/simonw/datasette/issues/1023,712604541,MDEyOklzc3VlQ29tbWVudDcxMjYwNDU0MQ==,9599,simonw,2020-10-20T05:39:44Z,2020-10-20T05:39:44Z,OWNER,Here's the alpha with most of this work ready for people to preview: https://github.com/simonw/datasette/releases/tag/0.51a0,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722673818,Fix issues relating to base_url, https://github.com/simonw/datasette/issues/838#issuecomment-712604364,https://api.github.com/repos/simonw/datasette/issues/838,712604364,MDEyOklzc3VlQ29tbWVudDcxMjYwNDM2NA==,9599,simonw,2020-10-20T05:39:15Z,2020-10-20T05:39:15Z,OWNER,"OK, I've made a ton of improvements to how the `base_url` setting works - see tickets linked from #1023. I've just pushed out an alpha release with those changes in it: https://github.com/simonw/datasette/releases/tag/0.51a0 @tsibley @tballison @ChristopherWilks I'd really appreciate your help testing this alpha! You can install it with: pip install datasette==0.51a0 It should work with just `ProxyPass`, without needing the `ProxyPassReverse` setting.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",637395097,Incorrect URLs when served behind a proxy with base_url set, https://github.com/simonw/datasette/issues/865#issuecomment-712597762,https://api.github.com/repos/simonw/datasette/issues/865,712597762,MDEyOklzc3VlQ29tbWVudDcxMjU5Nzc2Mg==,9599,simonw,2020-10-20T05:22:59Z,2020-10-20T05:22:59Z,OWNER,"OK, this is definitely working now.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",644582921,"base_url doesn't seem to work when adding criteria and clicking ""apply""", https://github.com/simonw/datasette/issues/1025#issuecomment-712593790,https://api.github.com/repos/simonw/datasette/issues/1025,712593790,MDEyOklzc3VlQ29tbWVudDcxMjU5Mzc5MA==,9599,simonw,2020-10-20T05:12:36Z,2020-10-20T05:12:36Z,OWNER,"I'm going to leave the cookies code setting cookies to default to the `""/""` top level.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722724086,"Fix last remaining links to ""/"" that do not respect base_url", https://github.com/simonw/datasette/issues/782#issuecomment-712590398,https://api.github.com/repos/simonw/datasette/issues/782,712590398,MDEyOklzc3VlQ29tbWVudDcxMjU5MDM5OA==,9599,simonw,2020-10-20T05:03:46Z,2020-10-20T05:04:09Z,OWNER,"OK, https://latest-with-plugins.datasette.io/ is running that now - e.g. https://latest-with-plugins.datasette.io/fixtures/roadside_attractions.json-preview or https://latest-with-plugins.datasette.io/fixtures/compound_three_primary_keys.json-preview ```json { ""rows"": [ { ""pk"": 1, ""name"": ""The Mystery Spot"", ""address"": ""465 Mystery Spot Road, Santa Cruz, CA 95065"", ""latitude"": 37.0167, ""longitude"": -122.0024 }, { ""pk"": 2, ""name"": ""Winchester Mystery House"", ""address"": ""525 South Winchester Boulevard, San Jose, CA 95128"", ""latitude"": 37.3184, ""longitude"": -121.9511 }, { ""pk"": 3, ""name"": ""Burlingame Museum of PEZ Memorabilia"", ""address"": ""214 California Drive, Burlingame, CA 94010"", ""latitude"": 37.5793, ""longitude"": -122.3442 }, { ""pk"": 4, ""name"": ""Bigfoot Discovery Museum"", ""address"": ""5497 Highway 9, Felton, CA 95018"", ""latitude"": 37.0414, ""longitude"": -122.0725 } ], ""total"": 4, ""next_url"": null } ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format, https://github.com/simonw/datasette/issues/782#issuecomment-712585921,https://api.github.com/repos/simonw/datasette/issues/782,712585921,MDEyOklzc3VlQ29tbWVudDcxMjU4NTkyMQ==,9599,simonw,2020-10-20T04:48:01Z,2020-10-20T04:48:01Z,OWNER,I'll update `datasette-json-preview` with that now.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format, https://github.com/simonw/datasette/issues/782#issuecomment-712585687,https://api.github.com/repos/simonw/datasette/issues/782,712585687,MDEyOklzc3VlQ29tbWVudDcxMjU4NTY4Nw==,9599,simonw,2020-10-20T04:47:02Z,2020-10-20T04:47:12Z,OWNER,"Great point about CORS, I hadn't considered that. I think I'm going to keep the `Link:` header (added in #1014) because I quite enjoy using it with GitHub and WordPress, but I'm not going to have it be the default way of doing pagination. For the default shape I'm now leaning towards this: ```json { ""total"": 36, ""rows"": [{""id"": 1, ""name"": ""Cleo""}], ""next_url"": ""https://latest-with-plugins.datasette.io/fixtures/facetable.json?_next=5"" } ``` So three keys: `total`, `rows` and `next_url`. Then extra keys can be added using `?_extra=` with various named bundles.","{""total_count"": 3, ""+1"": 3, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format, https://github.com/simonw/datasette/issues/1034#issuecomment-712582699,https://api.github.com/repos/simonw/datasette/issues/1034,712582699,MDEyOklzc3VlQ29tbWVudDcxMjU4MjY5OQ==,9599,simonw,2020-10-20T04:36:04Z,2020-10-20T04:36:14Z,OWNER,Asked for ideas on Twitter: https://twitter.com/simonw/status/1318409558805467136,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725184645,Better way of representing binary data in .csv output, https://github.com/simonw/datasette/issues/1034#issuecomment-712581994,https://api.github.com/repos/simonw/datasette/issues/1034,712581994,MDEyOklzc3VlQ29tbWVudDcxMjU4MTk5NA==,9599,simonw,2020-10-20T04:33:28Z,2020-10-20T04:33:28Z,OWNER,"The [datasette-render-binary](https://github.com/simonw/datasette-render-binary) plugin does this, which I really like - but without the different coloured fonts I'm not sure how readable it would be as just plain text: ![image](https://user-images.githubusercontent.com/9599/96540435-9c125f00-1252-11eb-85aa-5fc8d0e63728.png) Really the goal here is to find the most human-friendly option, so that people looking at the output have a vague idea what's going on. That's why I'm not leaping at the chance to use base64.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725184645,Better way of representing binary data in .csv output, https://github.com/simonw/datasette/issues/1034#issuecomment-712580976,https://api.github.com/repos/simonw/datasette/issues/1034,712580976,MDEyOklzc3VlQ29tbWVudDcxMjU4MDk3Ng==,9599,simonw,2020-10-20T04:29:23Z,2020-10-20T04:29:23Z,OWNER,Most obvious option is base64. Any other potential solutions I'm missing?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725184645,Better way of representing binary data in .csv output, https://github.com/simonw/datasette/issues/1025#issuecomment-712579674,https://api.github.com/repos/simonw/datasette/issues/1025,712579674,MDEyOklzc3VlQ29tbWVudDcxMjU3OTY3NA==,9599,simonw,2020-10-20T04:24:10Z,2020-10-20T04:24:10Z,OWNER,"Changed my mind, `error.html` needs access to `urls` in order to link to its CSS file. Passing it after all (it already got passed `ds.config(""base_url"")` so `ds` was available previously).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722724086,"Fix last remaining links to ""/"" that do not respect base_url", https://github.com/simonw/datasette/issues/782#issuecomment-712569695,https://api.github.com/repos/simonw/datasette/issues/782,712569695,MDEyOklzc3VlQ29tbWVudDcxMjU2OTY5NQ==,222245,carlmjohnson,2020-10-20T03:45:48Z,2020-10-20T03:46:14Z,NONE,"I vote against headers. It has a lot of strikes against it: poor discoverability, new developers often don’t know how to use them, makes CORS harder, makes it hard to use eg with JQ, needs ad hoc specification for each bit of metadata, etc. The only advantage of headers is that you don’t need to do .rows, but that’s actually good as a data validation step anyway—if .rows is missing assume there’s an error and do your error handling path instead of parsing the rest.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",627794879,Redesign default .json format, https://github.com/simonw/datasette/issues/1025#issuecomment-712481127,https://api.github.com/repos/simonw/datasette/issues/1025,712481127,MDEyOklzc3VlQ29tbWVudDcxMjQ4MTEyNw==,9599,simonw,2020-10-19T22:40:37Z,2020-10-20T01:21:36Z,OWNER,Was blocked on #904 - now unblocked.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722724086,"Fix last remaining links to ""/"" that do not respect base_url", https://github.com/simonw/datasette/issues/1033#issuecomment-712529413,https://api.github.com/repos/simonw/datasette/issues/1033,712529413,MDEyOklzc3VlQ29tbWVudDcxMjUyOTQxMw==,9599,simonw,2020-10-20T01:21:12Z,2020-10-20T01:21:12Z,OWNER,Also refs #1023,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",725099777,datasette.urls.static_plugins(...) method, https://github.com/simonw/datasette/issues/1025#issuecomment-712525557,https://api.github.com/repos/simonw/datasette/issues/1025,712525557,MDEyOklzc3VlQ29tbWVudDcxMjUyNTU1Nw==,9599,simonw,2020-10-20T01:07:02Z,2020-10-20T01:07:02Z,OWNER,"I fixed the `queries.html` one. I'm not going to fix these two: ``` datasette/templates/error.html: home datasette/templates/patterns.html: home / ``` Because the `error.html` one does not get passed a context (which makes sense since an error has occurred) and the pattern portfolio doesn't need to link to anywhere in particular.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722724086,"Fix last remaining links to ""/"" that do not respect base_url", https://github.com/simonw/datasette/issues/904#issuecomment-712524699,https://api.github.com/repos/simonw/datasette/issues/904,712524699,MDEyOklzc3VlQ29tbWVudDcxMjUyNDY5OQ==,9599,simonw,2020-10-20T01:04:12Z,2020-10-20T01:04:12Z,OWNER,Documentation is https://docs.datasette.io/en/latest/writing_plugins.html#building-urls-within-plugins and https://docs.datasette.io/en/latest/internals.html#internals-datasette-urls,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",663228985,"datasette.urls.table() / .instance() / .database() methods for constructing URLs, also exposed to templates", https://github.com/simonw/datasette/issues/904#issuecomment-712483066,https://api.github.com/repos/simonw/datasette/issues/904,712483066,MDEyOklzc3VlQ29tbWVudDcxMjQ4MzA2Ng==,9599,simonw,2020-10-19T22:46:48Z,2020-10-19T22:46:48Z,OWNER,"OK, I'm committing to `datasette.urls.table()` and friends, which will be available to (and documented for use in) plugins and will be exposed to templates as `{{ urls.table(...) }}`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",663228985,"datasette.urls.table() / .instance() / .database() methods for constructing URLs, also exposed to templates", https://github.com/simonw/datasette/issues/1020#issuecomment-712482504,https://api.github.com/repos/simonw/datasette/issues/1020,712482504,MDEyOklzc3VlQ29tbWVudDcxMjQ4MjUwNA==,9599,simonw,2020-10-19T22:45:01Z,2020-10-19T22:45:01Z,OWNER,"I'm having trouble coming up with the syntax for this. Here's one option: ```python response = await datasette.client.get(path, request=request) ``` But this feels confusing to me. We're not using the `request=` argument as a request - we're using it as a source of some default request values (the cookies and incoming headers, but not the path). We're essentially combining that request with the other arguments passed to `.get()`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",721068929,Method for datasette.client() to forward on authentication, https://github.com/simonw/datasette/issues/1020#issuecomment-712482015,https://api.github.com/repos/simonw/datasette/issues/1020,712482015,MDEyOklzc3VlQ29tbWVudDcxMjQ4MjAxNQ==,9599,simonw,2020-10-19T22:43:24Z,2020-10-19T22:43:24Z,OWNER,"... unless I want to support authentication mechanisms that work based on incoming IP address instead, in which case there's an argument for copying more over from the incoming request. Tailscale is a good example of a system where authentication based on IP address can actually work well, so this is worth doing. Also, there might be authentication mechanisms which work by setting a custom header on the incoming request (not to mention the `Authorization` header).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",721068929,Method for datasette.client() to forward on authentication, https://github.com/simonw/datasette/issues/1020#issuecomment-712481568,https://api.github.com/repos/simonw/datasette/issues/1020,712481568,MDEyOklzc3VlQ29tbWVudDcxMjQ4MTU2OA==,9599,simonw,2020-10-19T22:41:59Z,2020-10-19T22:41:59Z,OWNER,"It turns out this works just fine: ```python response = await datasette.client.get(path, cookies=request.cookies) ``` So I don't need a mechanism for this. I'm going to add this to the documentation instead.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",721068929,Method for datasette.client() to forward on authentication, https://github.com/simonw/datasette/issues/1028#issuecomment-712480866,https://api.github.com/repos/simonw/datasette/issues/1028,712480866,MDEyOklzc3VlQ29tbWVudDcxMjQ4MDg2Ng==,9599,simonw,2020-10-19T22:39:51Z,2020-10-19T22:39:51Z,OWNER,Documentation: https://docs.datasette.io/en/latest/spatialite.html,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",723803777,--load-extension=spatialite shortcut, https://github.com/simonw/datasette/issues/1032#issuecomment-712397537,https://api.github.com/repos/simonw/datasette/issues/1032,712397537,MDEyOklzc3VlQ29tbWVudDcxMjM5NzUzNw==,236498,saulpw,2020-10-19T19:37:55Z,2020-10-19T19:37:55Z,NONE,"python-dateutil is awesome, but it can only guess at one date at a time. So if you have a column of dates that are (presumably) in the same format, it can't use the full set of dates to deduce the format. Also, once it has parsed a date, you can't get the format it used, whether to parse or render other dates. These limitations prevent it from being a silver bullet for date parsing, though they're not enough for me to stop using it!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",724878151,Bring date parsing into Datasette core, https://github.com/simonw/datasette/issues/1032#issuecomment-712367285,https://api.github.com/repos/simonw/datasette/issues/1032,712367285,MDEyOklzc3VlQ29tbWVudDcxMjM2NzI4NQ==,9599,simonw,2020-10-19T18:39:32Z,2020-10-19T18:39:32Z,OWNER,https://github.com/digital-land/brownfield-land-collection/blob/a09ddf9960a6af59e72dc02448f7b645e59bf227/bin/harmonise.py#L217-L247 is a beautiful example of this problem.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",724878151,Bring date parsing into Datasette core, https://github.com/simonw/datasette/issues/1032#issuecomment-712365439,https://api.github.com/repos/simonw/datasette/issues/1032,712365439,MDEyOklzc3VlQ29tbWVudDcxMjM2NTQzOQ==,9599,simonw,2020-10-19T18:35:50Z,2020-10-19T18:37:57Z,OWNER,"Maybe I don't need to add `dateutil` as a dependency here? `pendulum` and `arrow` both depend on it so it's pretty deeply embedded in the Python date ecosystem. Adding it as a dependency seems reasonable.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",724878151,Bring date parsing into Datasette core, https://github.com/simonw/datasette/issues/1032#issuecomment-712365236,https://api.github.com/repos/simonw/datasette/issues/1032,712365236,MDEyOklzc3VlQ29tbWVudDcxMjM2NTIzNg==,9599,simonw,2020-10-19T18:35:25Z,2020-10-19T18:35:25Z,OWNER,"Since I'm dealing with tables of data I usually have a whole column of examples, so heuristics that check for numbers-greater-than-12 could actually work well for many cases.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",724878151,Bring date parsing into Datasette core, https://github.com/simonw/datasette/issues/1032#issuecomment-712364532,https://api.github.com/repos/simonw/datasette/issues/1032,712364532,MDEyOklzc3VlQ29tbWVudDcxMjM2NDUzMg==,9599,simonw,2020-10-19T18:34:10Z,2020-10-19T18:34:10Z,OWNER,Lots of great example date data in https://biglocal.datasettes.com/COVID_WARN_Notices,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",724878151,Bring date parsing into Datasette core, https://github.com/simonw/datasette/issues/1032#issuecomment-712364317,https://api.github.com/repos/simonw/datasette/issues/1032,712364317,MDEyOklzc3VlQ29tbWVudDcxMjM2NDMxNw==,9599,simonw,2020-10-19T18:33:42Z,2020-10-19T18:33:42Z,OWNER,"Related challenge: timezones. I think I'll punt on those for the moment and just concentrate on dates, but I should keep them in mind.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",724878151,Bring date parsing into Datasette core, https://github.com/simonw/datasette/issues/1032#issuecomment-712363825,https://api.github.com/repos/simonw/datasette/issues/1032,712363825,MDEyOklzc3VlQ29tbWVudDcxMjM2MzgyNQ==,9599,simonw,2020-10-19T18:32:43Z,2020-10-19T18:32:43Z,OWNER,"Horrible thought: I bet there is data out there that uses more than one date format in the same table! So this needs to be exposed as a visible per-column setting. Column action menu can help here, although it's not yet the full solution because it isn't yet visible on mobile.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",724878151,Bring date parsing into Datasette core, https://github.com/simonw/datasette/issues/1032#issuecomment-712363344,https://api.github.com/repos/simonw/datasette/issues/1032,712363344,MDEyOklzc3VlQ29tbWVudDcxMjM2MzM0NA==,9599,simonw,2020-10-19T18:31:46Z,2020-10-19T18:31:46Z,OWNER,"As always with dates, the challenge is America. `mm/dd/yy` is indistinguishable from the more sensible `dd/mm/yy`. It's crucially important to visibly tell people which format you assumed, otherwise all kinds of weird invisible bugs can manifest.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",724878151,Bring date parsing into Datasette core, https://github.com/simonw/datasette/issues/904#issuecomment-712355877,https://api.github.com/repos/simonw/datasette/issues/904,712355877,MDEyOklzc3VlQ29tbWVudDcxMjM1NTg3Nw==,9599,simonw,2020-10-19T18:17:22Z,2020-10-19T18:17:22Z,OWNER,I quite like `datasette.urls.table()` but I'm not sure why.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",663228985,"datasette.urls.table() / .instance() / .database() methods for constructing URLs, also exposed to templates", https://github.com/simonw/datasette/issues/904#issuecomment-712355706,https://api.github.com/repos/simonw/datasette/issues/904,712355706,MDEyOklzc3VlQ29tbWVudDcxMjM1NTcwNg==,9599,simonw,2020-10-19T18:17:03Z,2020-10-19T18:17:03Z,OWNER,"Options: - `datasette.urls.instance()` (or `datasette.urls.root()`), `datasette.urls.table()` etc - `datasette.url_for.instance()`... - `datasette.url.instance()`... - `datasette.root_url()`, `datasette.table_url()`...","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",663228985,"datasette.urls.table() / .instance() / .database() methods for constructing URLs, also exposed to templates", https://github.com/simonw/datasette/issues/904#issuecomment-712354600,https://api.github.com/repos/simonw/datasette/issues/904,712354600,MDEyOklzc3VlQ29tbWVudDcxMjM1NDYwMA==,9599,simonw,2020-10-19T18:15:03Z,2020-10-19T18:15:39Z,OWNER,"Related: #1026 (How should datasette.client interact with base_url) Also this comment from https://github.com/simonw/datasette/issues/943#issuecomment-675752436 > One thing to consider here: Datasette's table and database name escaping rules can be a little bit convoluted. > > If a plugin wants to get back the first five rows of a table, it will need to construct a URL `/dbname/tablename?_size=5` - but it will need to know how to turn the database and table names into the correctly escaped `dbname` and `tablename` values.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",663228985,"datasette.urls.table() / .instance() / .database() methods for constructing URLs, also exposed to templates", https://github.com/simonw/datasette/issues/904#issuecomment-712324077,https://api.github.com/repos/simonw/datasette/issues/904,712324077,MDEyOklzc3VlQ29tbWVudDcxMjMyNDA3Nw==,9599,simonw,2020-10-19T17:39:38Z,2020-10-19T17:39:38Z,OWNER,"If I do these methods I think this should be available on the `datasette` object too, as an internal API for plugins to use to construct redirects and suchlike.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",663228985,"datasette.urls.table() / .instance() / .database() methods for constructing URLs, also exposed to templates", https://github.com/simonw/datasette/issues/904#issuecomment-710487083,https://api.github.com/repos/simonw/datasette/issues/904,710487083,MDEyOklzc3VlQ29tbWVudDcxMDQ4NzA4Mw==,9599,simonw,2020-10-16T19:34:10Z,2020-10-19T17:39:04Z,OWNER,"Alternatively, I could expose a single object that knows how to construct all kinds of URLs. Something like this: ``` {{ urls.instance() }} {{ urls.database(database_name) }} {{ urls.table(database_name, table_name) }} etc ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",663228985,"datasette.urls.table() / .instance() / .database() methods for constructing URLs, also exposed to templates", https://github.com/simonw/datasette/issues/1027#issuecomment-712320103,https://api.github.com/repos/simonw/datasette/issues/1027,712320103,MDEyOklzc3VlQ29tbWVudDcxMjMyMDEwMw==,9599,simonw,2020-10-19T17:35:04Z,2020-10-19T17:35:04Z,OWNER,Still need to configure proxying though. https://www.netnea.com/cms/apache-tutorial-9_setting-up-a-reverse-proxy/,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722758132,Add documentation on serving Datasette behind a proxy using base_url, https://github.com/simonw/datasette/issues/991#issuecomment-712317638,https://api.github.com/repos/simonw/datasette/issues/991,712317638,MDEyOklzc3VlQ29tbWVudDcxMjMxNzYzOA==,9599,simonw,2020-10-19T17:30:56Z,2020-10-19T17:30:56Z,OWNER,https://biglocal.datasettes.com/ is one of my larger Datasettes in terms of number of databases.,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",714377268,Redesign application homepage, https://github.com/dogsheep/dogsheep-beta/issues/29#issuecomment-712266834,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/29,712266834,MDEyOklzc3VlQ29tbWVudDcxMjI2NjgzNA==,9599,simonw,2020-10-19T16:01:23Z,2020-10-19T16:01:23Z,MEMBER,Might just be a documented pattern for how to configure this in YAML templates.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",724759588,Add search highlighting snippets, https://github.com/simonw/datasette/pull/1030#issuecomment-711407607,https://api.github.com/repos/simonw/datasette/issues/1030,711407607,MDEyOklzc3VlQ29tbWVudDcxMTQwNzYwNw==,22429695,codecov[bot],2020-10-18T19:31:31Z,2020-10-19T08:01:51Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1030?src=pr&el=h1) Report > Merging [#1030](https://codecov.io/gh/simonw/datasette/pull/1030?src=pr&el=desc) into [main](https://codecov.io/gh/simonw/datasette/commit/568bd7bbf590861687db8c318f3d8cfcd1dfb47a?el=desc) will **decrease** coverage by `0.10%`. > The diff coverage is `68.75%`. [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1030/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1)](https://codecov.io/gh/simonw/datasette/pull/1030?src=pr&el=tree) ```diff @@ Coverage Diff @@ ## main #1030 +/- ## ========================================== - Coverage 84.63% 84.53% -0.11% ========================================== Files 28 28 Lines 3892 3905 +13 ========================================== + Hits 3294 3301 +7 - Misses 598 604 +6 ``` | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1030?src=pr&el=tree) | Coverage Δ | | |---|---|---| | [datasette/utils/\_\_init\_\_.py](https://codecov.io/gh/simonw/datasette/pull/1030/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3V0aWxzL19faW5pdF9fLnB5) | `93.35% <68.75%> (-0.79%)` | :arrow_down: | | [datasette/views/index.py](https://codecov.io/gh/simonw/datasette/pull/1030/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3ZpZXdzL2luZGV4LnB5) | `96.49% <0.00%> (-1.76%)` | :arrow_down: | ------ [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1030?src=pr&el=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1030?src=pr&el=footer). Last update [568bd7b...e082533](https://codecov.io/gh/simonw/datasette/pull/1030?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",723982480,Make `package` command deal with a configuration directory argument, https://github.com/simonw/datasette/pull/1031#issuecomment-711792622,https://api.github.com/repos/simonw/datasette/issues/1031,711792622,MDEyOklzc3VlQ29tbWVudDcxMTc5MjYyMg==,22429695,codecov[bot],2020-10-19T07:57:17Z,2020-10-19T07:57:17Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/1031?src=pr&el=h1) Report > Merging [#1031](https://codecov.io/gh/simonw/datasette/pull/1031?src=pr&el=desc) into [main](https://codecov.io/gh/simonw/datasette/commit/568bd7bbf590861687db8c318f3d8cfcd1dfb47a?el=desc) will **decrease** coverage by `0.02%`. > The diff coverage is `n/a`. [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1031/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1)](https://codecov.io/gh/simonw/datasette/pull/1031?src=pr&el=tree) ```diff @@ Coverage Diff @@ ## main #1031 +/- ## ========================================== - Coverage 84.63% 84.60% -0.03% ========================================== Files 28 28 Lines 3892 3892 ========================================== - Hits 3294 3293 -1 - Misses 598 599 +1 ``` | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1031?src=pr&el=tree) | Coverage Δ | | |---|---|---| | [datasette/cli.py](https://codecov.io/gh/simonw/datasette/pull/1031/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL2NsaS5weQ==) | `74.22% <ø> (ø)` | | | [datasette/views/index.py](https://codecov.io/gh/simonw/datasette/pull/1031/diff?src=pr&el=tree#diff-ZGF0YXNldHRlL3ZpZXdzL2luZGV4LnB5) | `96.49% <0.00%> (-1.76%)` | :arrow_down: | ------ [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1031?src=pr&el=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1031?src=pr&el=footer). Last update [568bd7b...7e7eaa4](https://codecov.io/gh/simonw/datasette/pull/1031?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",724369025,Fallback to databases in inspect-data.json when no -i options are passed, https://github.com/dogsheep/github-to-sqlite/issues/50#issuecomment-711569063,https://api.github.com/repos/dogsheep/github-to-sqlite/issues/50,711569063,MDEyOklzc3VlQ29tbWVudDcxMTU2OTA2Mw==,9599,simonw,2020-10-19T05:01:29Z,2020-10-19T05:01:29Z,MEMBER,"Demo of `--accept`: github-to-sqlite get /repos/simonw/datasette/readme --accept 'application/vnd.github.VERSION.html' ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",703218756,Commands for making authenticated API calls, https://github.com/dogsheep/dogsheep-beta/issues/28#issuecomment-711089647,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/28,711089647,MDEyOklzc3VlQ29tbWVudDcxMTA4OTY0Nw==,9599,simonw,2020-10-17T22:43:13Z,2020-10-17T22:43:13Z,MEMBER,"Since my personal Dogsheep uses Datasette authentication, I'm going to need to pass through cookies. https://github.com/simonw/datasette/issues/1020 will solve that in the future but for now I need to solve it explicitly.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",723861683,Switch to using datasette.client, https://github.com/dogsheep/healthkit-to-sqlite/issues/11#issuecomment-711083698,https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/11,711083698,MDEyOklzc3VlQ29tbWVudDcxMTA4MzY5OA==,572,jarib,2020-10-17T21:39:15Z,2020-10-17T21:39:15Z,NONE,Nice! Works perfectly. Thanks for the quick response and great tooling in general.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",723838331,export.xml file name varies with different language settings, https://github.com/dogsheep/healthkit-to-sqlite/issues/11#issuecomment-711081703,https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/11,711081703,MDEyOklzc3VlQ29tbWVudDcxMTA4MTcwMw==,9599,simonw,2020-10-17T21:18:35Z,2020-10-17T21:18:35Z,MEMBER,"OK, if you upgrade to the just-released 1.0 this should work (it worked against my Spanish export).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",723838331,export.xml file name varies with different language settings, https://github.com/dogsheep/healthkit-to-sqlite/issues/11#issuecomment-711079760,https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/11,711079760,MDEyOklzc3VlQ29tbWVudDcxMTA3OTc2MA==,9599,simonw,2020-10-17T21:00:05Z,2020-10-17T21:00:05Z,MEMBER,Checking for either ` "" TEMPLATE = """"""
{}
"""""".strip() EN_MEDIA_SCRIPT = """""" Array.from(document.querySelectorAll('en-media')).forEach(el => { let hash = el.getAttribute('hash'); let type = el.getAttribute('type'); let path = `/evernote/resources_data/${hash}.json?_shape=array`; fetch(path).then(r => r.json()).then(rows => { let b64 = rows[0].data.encoded; let data = `data:${type};base64,${b64}`; el.innerHTML = ``; }); }); """""" @hookimpl def render_cell(value, table): if not table: # Don't render content from arbitrary SQL queries, could be XSS hole return if not value or not isinstance(value, str): return value = value.strip() if value.startswith(START) and value.endswith(END): trimmed = value[len(START) : -len(END)] trimmed = trimmed.split("">"", 1)[1] # Replace those horrible double newlines trimmed = trimmed.replace(""

"", ""
"") return jinja2.Markup(TEMPLATE.format(trimmed)) @hookimpl def extra_body_script(): return EN_MEDIA_SCRIPT ``` It works! It does however demonstrate that Evernote's ""clip this webpage"" feature means there is a LOT of weird HTML that can get into a note. It looks like they've filtered out the scripts but I wouldn't bet on it - they certainly don't filter out many of the inline styles. So running Bleach is almost certainly a good idea.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",718938889,Figure out how to display images from tags inline in Datasette, https://github.com/simonw/sqlite-utils/issues/49#issuecomment-710461468,https://api.github.com/repos/simonw/sqlite-utils/issues/49,710461468,MDEyOklzc3VlQ29tbWVudDcxMDQ2MTQ2OA==,9599,simonw,2020-10-16T19:18:19Z,2020-10-16T19:18:19Z,OWNER,"Reconsidering: #89 was a feature request that relates to this, so maybe this is worth implementing after all.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",472115381,extracts= should support multiple-column extracts, https://github.com/simonw/sqlite-utils/issues/89#issuecomment-710460242,https://api.github.com/repos/simonw/sqlite-utils/issues/89,710460242,MDEyOklzc3VlQ29tbWVudDcxMDQ2MDI0Mg==,9599,simonw,2020-10-16T19:17:27Z,2020-10-16T19:17:50Z,OWNER,"I came up with potential syntax for that here: https://github.com/simonw/sqlite-utils/issues/49#issuecomment-710393550 - based on how `table.extract(...)` works: ```python fresh_db.table(""tree"", extracts=[Extract( columns=(""CommonName"", ""LatinName""), table=""Species"", fk_column=""species_id"", rename={""CommonName"": ""name"", ""LatinName"": ""latin""} )]) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",573578548,Ability to customize columns used by extracts= feature, https://github.com/simonw/sqlite-utils/issues/187#issuecomment-710456981,https://api.github.com/repos/simonw/sqlite-utils/issues/187,710456981,MDEyOklzc3VlQ29tbWVudDcxMDQ1Njk4MQ==,9599,simonw,2020-10-16T19:15:13Z,2020-10-16T19:15:13Z,OWNER,This is a duplicate of #79.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",723460107,Maybe: Utility method / CLI tool for initializing SpatiaLite, https://github.com/simonw/sqlite-utils/issues/187#issuecomment-710440853,https://api.github.com/repos/simonw/sqlite-utils/issues/187,710440853,MDEyOklzc3VlQ29tbWVudDcxMDQ0MDg1Mw==,9599,simonw,2020-10-16T19:04:19Z,2020-10-16T19:04:19Z,OWNER,I split this off from #136.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",723460107,Maybe: Utility method / CLI tool for initializing SpatiaLite, https://github.com/simonw/sqlite-utils/issues/136#issuecomment-710428802,https://api.github.com/repos/simonw/sqlite-utils/issues/136,710428802,MDEyOklzc3VlQ29tbWVudDcxMDQyODgwMg==,9599,simonw,2020-10-16T18:56:47Z,2020-10-16T18:56:47Z,OWNER,"To keep the code cleaner, I'm tempted to support this instead: --load-extension=spatialite Where `spatialite` is a special shortcut value that triggers a search for that module in known locations. Users could still load a module in a file called `spatialite` in the current directory using: --load-extension=./spatialite In fact, `--load-extension=spatialite` could handle that case too by always checking for a file called `spatialite` before attempting to search for it in known locations.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",683812642,--load-extension=spatialite shortcut option, https://github.com/simonw/sqlite-utils/issues/69#issuecomment-710405658,https://api.github.com/repos/simonw/sqlite-utils/issues/69,710405658,MDEyOklzc3VlQ29tbWVudDcxMDQwNTY1OA==,9599,simonw,2020-10-16T18:42:48Z,2020-10-16T18:42:48Z,OWNER,"Did some work on this for #134 in 7e9aad7e1c09d1cf80d0b4d17d6157212a4b857d I still need to add `--load-extension` to other CLI methods, see #137. Closing this issue in favour of that one.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",534507142,Feature request: enable extensions loading, https://github.com/simonw/sqlite-utils/issues/48#issuecomment-710402331,https://api.github.com/repos/simonw/sqlite-utils/issues/48,710402331,MDEyOklzc3VlQ29tbWVudDcxMDQwMjMzMQ==,9599,simonw,2020-10-16T18:41:06Z,2020-10-16T18:41:06Z,OWNER,I could use this demo from JupyterCon 2020 https://gist.github.com/simonw/656c21b5800d5e4624dec9930f00e093,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",471818939,"Jupyter notebook demo of the library, launchable on Binder", https://github.com/simonw/sqlite-utils/issues/58#issuecomment-710399593,https://api.github.com/repos/simonw/sqlite-utils/issues/58,710399593,MDEyOklzc3VlQ29tbWVudDcxMDM5OTU5Mw==,9599,simonw,2020-10-16T18:39:31Z,2020-10-16T18:39:31Z,OWNER,I don't think this is valuable enough to justify adding to the library - especially since you can execute FTS search against views by joining to an FTS table built against an underlying table.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",488293926,Support enabling FTS on views, https://github.com/simonw/sqlite-utils/issues/49#issuecomment-710397574,https://api.github.com/repos/simonw/sqlite-utils/issues/49,710397574,MDEyOklzc3VlQ29tbWVudDcxMDM5NzU3NA==,9599,simonw,2020-10-16T18:38:21Z,2020-10-16T18:38:21Z,OWNER,"I'm not going to implement this. I'll leave `extract=...` as it is right now, suitable for quick simple single-column operations on input, but if users want to do something more complicated involving multiple columns they should use the `table.extract()` method after the initial insert instead.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",472115381,extracts= should support multiple-column extracts, https://github.com/simonw/sqlite-utils/issues/49#issuecomment-710395444,https://api.github.com/repos/simonw/sqlite-utils/issues/49,710395444,MDEyOklzc3VlQ29tbWVudDcxMDM5NTQ0NA==,9599,simonw,2020-10-16T18:37:10Z,2020-10-16T18:37:10Z,OWNER,"But this begins to feel too complicated, given that `table.extract()` can already be used to achieve the same thing.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",472115381,extracts= should support multiple-column extracts, https://github.com/simonw/sqlite-utils/issues/49#issuecomment-710393550,https://api.github.com/repos/simonw/sqlite-utils/issues/49,710393550,MDEyOklzc3VlQ29tbWVudDcxMDM5MzU1MA==,9599,simonw,2020-10-16T18:35:57Z,2020-10-16T18:36:39Z,OWNER,"If I want to support that most complicated example, I think the option to pass a `Extracts()` object to `extracts=` is the best way to do it: ```python fresh_db.table(""tree"", extracts=[Extract( columns=(""CommonName"", ""LatinName""), table=""Species"", fk_column=""species_id"", rename={""CommonName"": ""name"", ""LatinName"": ""latin""} )]) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",472115381,extracts= should support multiple-column extracts, https://github.com/simonw/sqlite-utils/issues/49#issuecomment-710390915,https://api.github.com/repos/simonw/sqlite-utils/issues/49,710390915,MDEyOklzc3VlQ29tbWVudDcxMDM5MDkxNQ==,9599,simonw,2020-10-16T18:34:26Z,2020-10-16T18:34:50Z,OWNER,"Here's the most complex example of `.extracts()`: ```python db[""Trees""].extract( [""CommonName"", ""LatinName""], table=""Species"", fk_column=""species_id"", rename={""CommonName"": ""name"", ""LatinName"": ""latin""} ) ``` Resulting in: ```sql CREATE TABLE [Species] ( [id] INTEGER PRIMARY KEY, [name] TEXT, [latin] TEXT ) ``` From https://sqlite-utils.readthedocs.io/en/stable/python-api.html#extracting-columns-into-a-separate-table","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",472115381,extracts= should support multiple-column extracts, https://github.com/simonw/sqlite-utils/issues/49#issuecomment-710364942,https://api.github.com/repos/simonw/sqlite-utils/issues/49,710364942,MDEyOklzc3VlQ29tbWVudDcxMDM2NDk0Mg==,9599,simonw,2020-10-16T18:18:48Z,2020-10-16T18:18:48Z,OWNER,"I think there is. It's a nice existing feature, and I don't think adding tuple support to it would be a huge lift.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",472115381,extracts= should support multiple-column extracts, https://github.com/simonw/sqlite-utils/issues/49#issuecomment-710363789,https://api.github.com/repos/simonw/sqlite-utils/issues/49,710363789,MDEyOklzc3VlQ29tbWVudDcxMDM2Mzc4OQ==,9599,simonw,2020-10-16T18:18:05Z,2020-10-16T18:18:05Z,OWNER,I wonder if there's value in extending the `extracts=` option at all given the existence of `table.extract()`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",472115381,extracts= should support multiple-column extracts, https://github.com/simonw/sqlite-utils/issues/49#issuecomment-710359724,https://api.github.com/repos/simonw/sqlite-utils/issues/49,710359724,MDEyOklzc3VlQ29tbWVudDcxMDM1OTcyNA==,9599,simonw,2020-10-16T18:15:31Z,2020-10-16T18:15:31Z,OWNER,"Using a tuple would work: ```python fresh_db.table(""tree"", extracts=[(""common_name"", ""latin_name"")]) ``` Or to define a custom name: ```python fresh_db.table(""tree"", extracts={(""common_name"", ""latin_name""): ""names""}) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",472115381,extracts= should support multiple-column extracts, https://github.com/simonw/sqlite-utils/issues/49#issuecomment-710346830,https://api.github.com/repos/simonw/sqlite-utils/issues/49,710346830,MDEyOklzc3VlQ29tbWVudDcxMDM0NjgzMA==,9599,simonw,2020-10-16T18:08:52Z,2020-10-16T18:09:21Z,OWNER,"The new `.extract()` method can handle multiple columns: https://github.com/simonw/sqlite-utils/blob/2c541fac352632e23e40b0d21e3f233f7a744a57/tests/test_extract.py#L70-L87","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",472115381,extracts= should support multiple-column extracts, https://github.com/simonw/sqlite-utils/issues/182#issuecomment-710258736,https://api.github.com/repos/simonw/sqlite-utils/issues/182,710258736,MDEyOklzc3VlQ29tbWVudDcxMDI1ODczNg==,9599,simonw,2020-10-16T17:20:41Z,2020-10-16T17:20:41Z,OWNER,Documentation: https://sqlite-utils.readthedocs.io/en/latest/cli.html#inserting-csv-or-tsv-data,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",711649325,"Better handling of encodings other than utf-8 for ""sqlite-utils insert""", https://github.com/simonw/sqlite-utils/issues/186#issuecomment-710198162,https://api.github.com/repos/simonw/sqlite-utils/issues/186,710198162,MDEyOklzc3VlQ29tbWVudDcxMDE5ODE2Mg==,9599,simonw,2020-10-16T16:41:00Z,2020-10-16T16:41:00Z,OWNER,"Failing test: ```python def test_extract_null_values(fresh_db): fresh_db[""species""].insert({""id"": 1, ""species"": ""Wolf""}, pk=""id"") fresh_db[""individuals""].insert_all( [ {""id"": 10, ""name"": ""Terriana"", ""species"": ""Fox""}, {""id"": 11, ""name"": ""Spenidorm"", ""species"": None}, {""id"": 12, ""name"": ""Grantheim"", ""species"": ""Wolf""}, {""id"": 13, ""name"": ""Turnutopia"", ""species"": None}, {""id"": 14, ""name"": ""Wargal"", ""species"": ""Wolf""}, ], pk=""id"", ) fresh_db[""individuals""].extract(""species"") assert fresh_db[""species""].schema == ( ""CREATE TABLE [species] (\n"" "" [id] INTEGER PRIMARY KEY,\n"" "" [species] TEXT\n"" "")"" ) assert fresh_db[""individuals""].schema == ( 'CREATE TABLE ""individuals"" (\n' "" [id] INTEGER PRIMARY KEY,\n"" "" [name] TEXT,\n"" "" [species_id] INTEGER,\n"" "" FOREIGN KEY(species_id) REFERENCES species(id)\n"" "")"" ) assert list(fresh_db[""species""].rows) == [ {""id"": 1, ""species"": ""Wolf""}, {""id"": 2, ""species"": ""Fox""}, ] assert list(fresh_db[""individuals""].rows) == [ {""id"": 10, ""name"": ""Terriana"", ""species_id"": 2}, {""id"": 11, ""name"": ""Spenidorm"", ""species_id"": None}, {""id"": 12, ""name"": ""Grantheim"", ""species_id"": 1}, {""id"": 13, ""name"": ""Turnutopia"", ""species_id"": None}, {""id"": 14, ""name"": ""Wargal"", ""species_id"": 1}, ] ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722816436,.extract() shouldn't extract null values, https://github.com/simonw/sqlite-utils/issues/182#issuecomment-710178871,https://api.github.com/repos/simonw/sqlite-utils/issues/182,710178871,MDEyOklzc3VlQ29tbWVudDcxMDE3ODg3MQ==,9599,simonw,2020-10-16T16:27:39Z,2020-10-16T16:28:14Z,OWNER,"The file is opened for me by `click.File()`, which also handles things like `-` for stdin. But i neee to be able to switch the encoding used to read from that based on the `--encoding` option. I think the way to do that is to open the file in binary mode and then wrap it in a codec reader: ```python fp = codecs.getreader(encoding)(fp) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",711649325,"Better handling of encodings other than utf-8 for ""sqlite-utils insert""", https://github.com/simonw/sqlite-utils/issues/186#issuecomment-709706260,https://api.github.com/repos/simonw/sqlite-utils/issues/186,709706260,MDEyOklzc3VlQ29tbWVudDcwOTcwNjI2MA==,9599,simonw,2020-10-16T03:17:02Z,2020-10-16T03:17:17Z,OWNER,Actually I think this should be an option to `.extract()` which controls if nulls are extracted or left alone. Maybe called `extract_nulls=True/False`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722816436,.extract() shouldn't extract null values, https://github.com/simonw/sqlite-utils/issues/186#issuecomment-709706065,https://api.github.com/repos/simonw/sqlite-utils/issues/186,709706065,MDEyOklzc3VlQ29tbWVudDcwOTcwNjA2NQ==,9599,simonw,2020-10-16T03:16:22Z,2020-10-16T03:16:22Z,OWNER,Either way I think I'm going to need to add some SQL which uses `where a = b or (a is null and b is null)`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722816436,.extract() shouldn't extract null values, https://github.com/simonw/sqlite-utils/issues/186#issuecomment-709705885,https://api.github.com/repos/simonw/sqlite-utils/issues/186,709705885,MDEyOklzc3VlQ29tbWVudDcwOTcwNTg4NQ==,9599,simonw,2020-10-16T03:15:39Z,2020-10-16T03:15:39Z,OWNER,The alternative solution here would be that a single `null` value DOES get extracted. To implement this I would need to add some logic that uses `is null` instead of `=`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722816436,.extract() shouldn't extract null values, https://github.com/simonw/sqlite-utils/issues/186#issuecomment-709705624,https://api.github.com/repos/simonw/sqlite-utils/issues/186,709705624,MDEyOklzc3VlQ29tbWVudDcwOTcwNTYyNA==,9599,simonw,2020-10-16T03:14:39Z,2020-10-16T03:14:39Z,OWNER,"How should this work with extractions covering multiple columns? If there's a single column then it makes sense that a `null` value would not be extracted into the lookup table, but would instead become stay as `null`. For a multiple column extraction, provided at least one of those columns is not null It should map to a record in the lookup table. Only if ALL of the extracted columns are null should the lookup value stay null.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722816436,.extract() shouldn't extract null values, https://github.com/simonw/datasette/issues/1027#issuecomment-709647525,https://api.github.com/repos/simonw/datasette/issues/1027,709647525,MDEyOklzc3VlQ29tbWVudDcwOTY0NzUyNQ==,9599,simonw,2020-10-15T23:49:51Z,2020-10-15T23:51:39Z,OWNER,"I'll install Apache on macOS to figure this out using https://formulae.brew.sh/formula/httpd `brew install httpd` output this at the end: ``` ==> httpd DocumentRoot is /usr/local/var/www. The default ports have been set in /usr/local/etc/httpd/httpd.conf to 8080 and in /usr/local/etc/httpd/extra/httpd-ssl.conf to 8443 so that httpd can run without sudo. To have launchd start httpd now and restart at login: brew services start httpd Or, if you don't want/need a background service you can just run: apachectl start ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722758132,Add documentation on serving Datasette behind a proxy using base_url, https://github.com/simonw/datasette/issues/1027#issuecomment-709646865,https://api.github.com/repos/simonw/datasette/issues/1027,709646865,MDEyOklzc3VlQ29tbWVudDcwOTY0Njg2NQ==,9599,simonw,2020-10-15T23:47:08Z,2020-10-15T23:47:08Z,OWNER,It should cover both nginx and Apache. nginx config is here: https://github.com/simonw/datasette/issues/1024#issuecomment-709598324,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722758132,Add documentation on serving Datasette behind a proxy using base_url, https://github.com/simonw/datasette/issues/1026#issuecomment-709636372,https://api.github.com/repos/simonw/datasette/issues/1026,709636372,MDEyOklzc3VlQ29tbWVudDcwOTYzNjM3Mg==,9599,simonw,2020-10-15T23:09:34Z,2020-10-15T23:09:34Z,OWNER,"I'm inclined to say that internal requests should ignore `base_url` - since that seems like the right thing for plugins that need to access default Datasette APIs. The one catch here is plugins that might want to proxy the current incoming URL for some reason - where that incoming `request.path` could include the `base_url`. Actually those should be fine - because it will have been stripped off earlier: https://github.com/simonw/datasette/blob/4f7c0ebd85ccd8c1853d7aa0147628f7c1b749cc/datasette/app.py#L963-L968","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722738988,How should datasette.client interact with base_url, https://github.com/simonw/datasette/issues/904#issuecomment-709635276,https://api.github.com/repos/simonw/datasette/issues/904,709635276,MDEyOklzc3VlQ29tbWVudDcwOTYzNTI3Ng==,9599,simonw,2020-10-15T23:05:54Z,2020-10-15T23:05:54Z,OWNER,Could have `instance_url()` take an optional path argument which is then turned into the correct path.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",663228985,"datasette.urls.table() / .instance() / .database() methods for constructing URLs, also exposed to templates", https://github.com/simonw/datasette/issues/904#issuecomment-709635021,https://api.github.com/repos/simonw/datasette/issues/904,709635021,MDEyOklzc3VlQ29tbWVudDcwOTYzNTAyMQ==,9599,simonw,2020-10-15T23:05:11Z,2020-10-15T23:05:11Z,OWNER,"I think this should be a family of functions: - `instance_url()` - the root URL of the instance (usually `/` unless `base_url` is set) - `database_url(database_name)` - already got this - `table_url(database_name, table_name)` - `row_url(database_name, table_name, row)` - not sure about this one. The idea would be for `row` to be correctly turned into a URL by introspecting the primary keys for that table, then pulling those values out of the SQLite `row` object. Might not be necessary though. I also need a way for plugins to link to e.g. `/-/configure-fts` - or even `/-/configure-fts/database-name/table-name`. What should that look like?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",663228985,"datasette.urls.table() / .instance() / .database() methods for constructing URLs, also exposed to templates", https://github.com/simonw/datasette/issues/904#issuecomment-709634261,https://api.github.com/repos/simonw/datasette/issues/904,709634261,MDEyOklzc3VlQ29tbWVudDcwOTYzNDI2MQ==,9599,simonw,2020-10-15T23:02:43Z,2020-10-15T23:02:43Z,OWNER,"Here's the current implementation of `database_url` - on the `BaseView` class, but only because it needs access to a `datasette` instance (to read `base_url`): https://github.com/simonw/datasette/blob/8f97b9b58e77f82fef1f10e9c9f6754b993544b6/datasette/views/base.py#L102-L108","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",663228985,"datasette.urls.table() / .instance() / .database() methods for constructing URLs, also exposed to templates", https://github.com/simonw/datasette/issues/904#issuecomment-709633823,https://api.github.com/repos/simonw/datasette/issues/904,709633823,MDEyOklzc3VlQ29tbWVudDcwOTYzMzgyMw==,9599,simonw,2020-10-15T23:01:13Z,2020-10-15T23:01:13Z,OWNER,Tracking ticket: #1023,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",663228985,"datasette.urls.table() / .instance() / .database() methods for constructing URLs, also exposed to templates", https://github.com/simonw/datasette/issues/988#issuecomment-709633762,https://api.github.com/repos/simonw/datasette/issues/988,709633762,MDEyOklzc3VlQ29tbWVudDcwOTYzMzc2Mg==,9599,simonw,2020-10-15T23:01:01Z,2020-10-15T23:01:01Z,OWNER,This is a dupe of https://github.com/simonw/datasette/issues/904,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",713209404,Mechanism for plugins to construct URLs that respect base_url, https://github.com/simonw/datasette/issues/865#issuecomment-709633080,https://api.github.com/repos/simonw/datasette/issues/865,709633080,MDEyOklzc3VlQ29tbWVudDcwOTYzMzA4MA==,9599,simonw,2020-10-15T22:58:51Z,2020-10-15T22:58:51Z,OWNER,"It looks like there are places where Datasette might return a redirect that doesn't take `base_url` into account - I'm planning on fixing those here, after which I think `ProxyPassReverse` should no longer be necessary. https://github.com/simonw/datasette/issues/1025#issuecomment-709632136","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 1, ""rocket"": 0, ""eyes"": 0}",644582921,"base_url doesn't seem to work when adding criteria and clicking ""apply""", https://github.com/simonw/datasette/issues/900#issuecomment-709632765,https://api.github.com/repos/simonw/datasette/issues/900,709632765,MDEyOklzc3VlQ29tbWVudDcwOTYzMjc2NQ==,9599,simonw,2020-10-15T22:57:55Z,2020-10-15T22:57:55Z,OWNER,"I believe this particular bug has been fixed, based on my testing here: https://github.com/simonw/datasette/issues/1024#issuecomment-709622973 Please re-open the ticket if you are still experiencing it. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",661605489,Some links don't honor base_url, https://github.com/simonw/datasette/issues/1025#issuecomment-709632314,https://api.github.com/repos/simonw/datasette/issues/1025,709632314,MDEyOklzc3VlQ29tbWVudDcwOTYzMjMxNA==,9599,simonw,2020-10-15T22:56:25Z,2020-10-15T22:56:34Z,OWNER,"That `utils/asgi.py` line is the default path for setting cookies. That should likely take `base_url` into account too: https://github.com/simonw/datasette/blob/4f7c0ebd85ccd8c1853d7aa0147628f7c1b749cc/datasette/utils/asgi.py#L331-L342","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722724086,"Fix last remaining links to ""/"" that do not respect base_url", https://github.com/simonw/datasette/issues/1025#issuecomment-709632136,https://api.github.com/repos/simonw/datasette/issues/1025,709632136,MDEyOklzc3VlQ29tbWVudDcwOTYzMjEzNg==,9599,simonw,2020-10-15T22:55:44Z,2020-10-15T22:55:44Z,OWNER,"It looks like there are also some generated redirect responses that don't take `base_url` into account: ``` datasette % git grep '""/' -- '*.py' ':(exclude)*test_*.py' ':(exclude)datasette/app.py' datasette/_version.py: for i in cfg.versionfile_source.split(""/""): datasette/utils/asgi.py: path=""/"", datasette/views/base.py: should_redirect = ""/{}-{}"".format(name, expected) datasette/views/base.py: should_redirect += ""/"" + urllib.parse.quote_plus(kwargs[""table""]) datasette/views/base.py: should_redirect += ""/"" + kwargs[""pk_path""] datasette/views/special.py: response = Response.redirect(""/"") datasette/views/special.py: return Response.redirect(""/"") datasette/views/special.py: response = Response.redirect(""/"") datasette/views/special.py: return Response.redirect(""/"") ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722724086,"Fix last remaining links to ""/"" that do not respect base_url", https://github.com/simonw/datasette/issues/1025#issuecomment-709629920,https://api.github.com/repos/simonw/datasette/issues/1025,709629920,MDEyOklzc3VlQ29tbWVudDcwOTYyOTkyMA==,9599,simonw,2020-10-15T22:48:20Z,2020-10-15T22:48:20Z,OWNER,"Also these: ``` datasette % git grep '""/' -- '*.html' ':(exclude)*/patterns.html' datasette/templates/allow_debug.html:
datasette/templates/base.html: datasette/templates/error.html: home datasette/templates/logout.html: datasette/templates/messages_debug.html: datasette/templates/query.html: home / ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722724086,"Fix last remaining links to ""/"" that do not respect base_url", https://github.com/simonw/datasette/issues/865#issuecomment-709626786,https://api.github.com/repos/simonw/datasette/issues/865,709626786,MDEyOklzc3VlQ29tbWVudDcwOTYyNjc4Ng==,9599,simonw,2020-10-15T22:38:38Z,2020-10-15T22:38:38Z,OWNER,"I managed to recreate proxying using `nginx` in #1024 - but I could not replicate this bug. I did NOT use `ProxyPassReverse` though. I think that may be what caused the problem. I'll add a section to the documentation about this shortly.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",644582921,"base_url doesn't seem to work when adding criteria and clicking ""apply""", https://github.com/simonw/datasette/issues/1024#issuecomment-709625063,https://api.github.com/repos/simonw/datasette/issues/1024,709625063,MDEyOklzc3VlQ29tbWVudDcwOTYyNTA2Mw==,9599,simonw,2020-10-15T22:33:22Z,2020-10-15T22:33:22Z,OWNER,"Of those errors... `http://localhost:8000/robots.txt` 404 is fine. `http://localhost:8000/datasette/%5C%22https://www.openstreetmap.org/copyright%5C%22` looks to me like a `wget` parsing bug where it got confused by this JavaScript: ``` window.DATASETTE_CLUSTER_MAP_TILE_LAYER_OPTIONS = {""maxZoom"": 19, ""detectRetina"": true, ""attribution"": ""© OpenStreetMap contributors""}; ``` `http://localhost:8000/-/static-plugins/datasette_cluster_map/datasette-cluster-map.js` is a real bug. It's a bug in `datasette-cluster-map` but also requires me to solve #988 - mechanism for plugins to construct URLs that obey `base_url`. I'm not sure why I'm getting a hit to `http://localhost:8000/` since I wouldn't expect to link to `/` anywhere.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722674708,Figure out how to run an environment that exercises the base_url proxy setting, https://github.com/simonw/datasette/issues/1024#issuecomment-709622973,https://api.github.com/repos/simonw/datasette/issues/1024,709622973,MDEyOklzc3VlQ29tbWVudDcwOTYyMjk3Mw==,9599,simonw,2020-10-15T22:27:31Z,2020-10-15T22:27:31Z,OWNER,"Here's how I tested it: ``` time wget -r 'http://localhost:8000/datasette/' 2>&1 | grep -i -C 5 ""failed\|error"" > /tmp/errors.txt ``` This wrote out any errors (plus context) to the `errors.txt` log - and reported that the full crawl took 33s. Here's what I got in `errors.txt`: ``` 0K . 71.6M=0s 2020-10-15 15:23:09 (71.6 MB/s) - ‘localhost:8000/datasette/index.html’ saved [1276] Loading robots.txt; please ignore errors. --2020-10-15 15:23:09-- http://localhost:8000/robots.txt Reusing existing connection to localhost:8000. HTTP request sent, awaiting response... 404 Not Found -- --2020-10-15 15:23:09-- http://localhost:8000/robots.txt Reusing existing connection to localhost:8000. HTTP request sent, awaiting response... 404 Not Found 2020-10-15 15:23:09 ERROR 404: Not Found. --2020-10-15 15:23:09-- http://localhost:8000/datasette/-/static/app.css?b576be Reusing existing connection to localhost:8000. HTTP request sent, awaiting response... 200 OK Length: 8563 (8.4K) [text/css] -- -- 2020-10-15 15:23:13 (7.90 MB/s) - ‘localhost:8000/datasette/fixtures/primary_key_multiple_columns_explicit_label.json?_shape=object’ saved [58] --2020-10-15 15:23:13-- http://localhost:8000/-/static-plugins/datasette_cluster_map/datasette-cluster-map.js Reusing existing connection to localhost:8000. HTTP request sent, awaiting response... 404 Not Found 2020-10-15 15:23:13 ERROR 404: Not Found. --2020-10-15 15:23:13-- http://localhost:8000/datasette/fixtures?sql=select+pk%2C+name%2C+address%2C+latitude%2C+longitude+from+roadside_attractions+order+by+pk+limit+101 Reusing existing connection to localhost:8000. HTTP request sent, awaiting response... 200 OK Length: unspecified [text/html] -- -- 2020-10-15 15:23:13 (84.3 MB/s) - ‘localhost:8000/datasette/fixtures/roadside_attractions.json?_shape=object’ saved [619] --2020-10-15 15:23:13-- http://localhost:8000/datasette/fixtures/%5C%22https://www.openstreetmap.org/copyright%5C%22 Reusing existing connection to localhost:8000. HTTP request sent, awaiting response... 404 Not Found 2020-10-15 15:23:13 ERROR 404: Not Found. --2020-10-15 15:23:13-- http://localhost:8000/datasette/fixtures?sql=select+pk%2C+text1%2C+text2%2C+%5Bname+with+.+and+spaces%5D+from+searchable+order+by+pk+limit+101 Reusing existing connection to localhost:8000. HTTP request sent, awaiting response... 200 OK Length: unspecified [text/html] -- -- 2020-10-15 15:23:14 (28.6 MB/s) - ‘localhost:8000/datasette/fixtures/searchable_view_configured_by_metadata.json?_shape=array&_nl=on’ saved [180] --2020-10-15 15:23:14-- http://localhost:8000/ Reusing existing connection to localhost:8000. HTTP request sent, awaiting response... 404 Not Found 2020-10-15 15:23:14 ERROR 404: Not Found. --2020-10-15 15:23:14-- http://localhost:8000/datasette/fixtures?sql=select+pk1%2C+pk2%2C+pk3%2C+content+from+compound_three_primary_keys+order+by+pk1%2C+pk2%2C+pk3+limit+101&_hide_sql=1 Reusing existing connection to localhost:8000. HTTP request sent, awaiting response... 200 OK Length: unspecified [text/html] -- -- 2020-10-15 15:23:21 (64.1 MB/s) - ‘localhost:8000/datasette/fixtures.csv?sql=select+pk,+name,+address,+latitude,+longitude+from+roadside_attractions+order+by+pk+limit+101&_size=max’ saved [403] --2020-10-15 15:23:21-- http://localhost:8000/datasette/%5C%22https://www.openstreetmap.org/copyright%5C%22 Reusing existing connection to localhost:8000. HTTP request sent, awaiting response... 404 Not Found 2020-10-15 15:23:21 ERROR 404: Not Found. --2020-10-15 15:23:21-- http://localhost:8000/datasette/fixtures?sql=select+pk%2C+name%2C+address%2C+latitude%2C+longitude+from+roadside_attractions+order+by+pk+desc+limit+101 Reusing existing connection to localhost:8000. HTTP request sent, awaiting response... 200 OK Length: unspecified [text/html] ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722674708,Figure out how to run an environment that exercises the base_url proxy setting, https://github.com/simonw/datasette/issues/1024#issuecomment-709600335,https://api.github.com/repos/simonw/datasette/issues/1024,709600335,MDEyOklzc3VlQ29tbWVudDcwOTYwMDMzNQ==,9599,simonw,2020-10-15T21:28:02Z,2020-10-15T22:25:43Z,OWNER,"This is working OK so far: I'll try crawling it with `wget -r` to see if I get any errors.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722674708,Figure out how to run an environment that exercises the base_url proxy setting, https://github.com/simonw/datasette/issues/1024#issuecomment-709598324,https://api.github.com/repos/simonw/datasette/issues/1024,709598324,MDEyOklzc3VlQ29tbWVudDcwOTU5ODMyNA==,9599,simonw,2020-10-15T21:23:33Z,2020-10-15T21:26:55Z,OWNER,"Combining these two examples, here's the config file I am going to use for this. I'll save this as `nginx.conf`: ``` daemon off; events { worker_connections 1024; } http { server { listen 8000; location /datasette { proxy_pass http://127.0.0.1:8001; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; } } } ``` Then start the server with: ``` nginx -p `pwd` -c `pwd`/nginx.conf ``` And start Datasette like this: ``` datasette fixtures.db --config base_url:/datasette/ ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722674708,Figure out how to run an environment that exercises the base_url proxy setting, https://github.com/simonw/datasette/issues/1024#issuecomment-709597589,https://api.github.com/repos/simonw/datasette/issues/1024,709597589,MDEyOklzc3VlQ29tbWVudDcwOTU5NzU4OQ==,9599,simonw,2020-10-15T21:21:53Z,2020-10-15T21:23:25Z,OWNER,"Here's a recipe for running nginx against a custom config file: https://gist.github.com/simonw/35f0ebf9c1d6df158759 ``` daemon off; events { worker_connections 1024; } http { access_log /dev/stdout; error_log /dev/stderr; types { text/html html htm shtml; text/css css; image/gif gif; image/jpeg jpeg jpg; application/javascript js; } server { listen 8002; index index.html; root app; } } ``` ``` nginx -p `pwd` -c `pwd`/nginx.conf ``` ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722674708,Figure out how to run an environment that exercises the base_url proxy setting, https://github.com/simonw/datasette/issues/1024#issuecomment-709595960,https://api.github.com/repos/simonw/datasette/issues/1024,709595960,MDEyOklzc3VlQ29tbWVudDcwOTU5NTk2MA==,9599,simonw,2020-10-15T21:18:14Z,2020-10-15T21:18:14Z,OWNER,Typing `nginx` starts it running as a daemon listening on port `http-alt` aka 8080. It uses the config file from ` /usr/local/etc/nginx/nginx.conf`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722674708,Figure out how to run an environment that exercises the base_url proxy setting, https://github.com/simonw/datasette/issues/1024#issuecomment-709590941,https://api.github.com/repos/simonw/datasette/issues/1024,709590941,MDEyOklzc3VlQ29tbWVudDcwOTU5MDk0MQ==,9599,simonw,2020-10-15T21:07:47Z,2020-10-15T21:07:47Z,OWNER,On macOS I ran `brew install nginx`. I'm going to try running it on port 8000 so I don't have to run it as root.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722674708,Figure out how to run an environment that exercises the base_url proxy setting, https://github.com/simonw/datasette/issues/1024#issuecomment-709590337,https://api.github.com/repos/simonw/datasette/issues/1024,709590337,MDEyOklzc3VlQ29tbWVudDcwOTU5MDMzNw==,9599,simonw,2020-10-15T21:06:24Z,2020-10-15T21:07:19Z,OWNER,"From https://stackoverflow.com/questions/32549684/nginx-proxy-and-remove-proxy-pass-prefix/32550251 it looks like the config I should use is: ``` server { listen 80; server_name example.com; location /datasette/ { proxy_pass http://127.0.0.1:8001; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; proxy_read_timeout 90; } } ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722674708,Figure out how to run an environment that exercises the base_url proxy setting, https://github.com/simonw/datasette/issues/1024#issuecomment-709589297,https://api.github.com/repos/simonw/datasette/issues/1024,709589297,MDEyOklzc3VlQ29tbWVudDcwOTU4OTI5Nw==,9599,simonw,2020-10-15T21:04:31Z,2020-10-15T21:04:31Z,OWNER,I think nginx or Apache would be the best tools for this. I'm inclined to try with nginx first since I know it better.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",722674708,Figure out how to run an environment that exercises the base_url proxy setting, https://github.com/simonw/datasette/issues/838#issuecomment-709588425,https://api.github.com/repos/simonw/datasette/issues/838,709588425,MDEyOklzc3VlQ29tbWVudDcwOTU4ODQyNQ==,9599,simonw,2020-10-15T21:02:37Z,2020-10-15T21:02:37Z,OWNER,Tracking ticket: #1023,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",637395097,Incorrect URLs when served behind a proxy with base_url set, https://github.com/simonw/datasette/issues/865#issuecomment-709588373,https://api.github.com/repos/simonw/datasette/issues/865,709588373,MDEyOklzc3VlQ29tbWVudDcwOTU4ODM3Mw==,9599,simonw,2020-10-15T21:02:31Z,2020-10-15T21:02:31Z,OWNER,Tracking ticket: #1023,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",644582921,"base_url doesn't seem to work when adding criteria and clicking ""apply""", https://github.com/simonw/datasette/issues/900#issuecomment-709588322,https://api.github.com/repos/simonw/datasette/issues/900,709588322,MDEyOklzc3VlQ29tbWVudDcwOTU4ODMyMg==,9599,simonw,2020-10-15T21:02:26Z,2020-10-15T21:02:26Z,OWNER,Tracking ticket: #1023,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",661605489,Some links don't honor base_url, https://github.com/simonw/datasette/issues/988#issuecomment-709588290,https://api.github.com/repos/simonw/datasette/issues/988,709588290,MDEyOklzc3VlQ29tbWVudDcwOTU4ODI5MA==,9599,simonw,2020-10-15T21:02:21Z,2020-10-15T21:02:21Z,OWNER,Tracking ticket: #1023,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",713209404,Mechanism for plugins to construct URLs that respect base_url, https://github.com/simonw/datasette/issues/894#issuecomment-709575818,https://api.github.com/repos/simonw/datasette/issues/894,709575818,MDEyOklzc3VlQ29tbWVudDcwOTU3NTgxOA==,9599,simonw,2020-10-15T20:35:03Z,2020-10-15T20:35:03Z,OWNER,"Prototype so far: ```diff diff --git a/datasette/views/table.py b/datasette/views/table.py index ea11a51..d61f8bd 100644 --- a/datasette/views/table.py +++ b/datasette/views/table.py @@ -497,17 +497,32 @@ class TableView(RowTableShared): if sort and sort_desc: raise DatasetteError(""Cannot use _sort and _sort_desc at the same time"") + def parse_sort(sort): + if ""~"" in sort: + if sort.endswith(""~default""): + col = sort.rsplit(""~"", 1)[0] + return col, escape_sqlite(col) + elif sort.endswith(""~numeric""): + col = sort.rsplit(""~"", 1)[0] + return col, ""cast(nullif({}, '') as real)"".format(escape_sqlite(col)) + else: + return sort, escape_sqlite(sort) + else: + return sort, escape_sqlite(sort) + if sort: - if sort not in sortable_columns: - raise DatasetteError(""Cannot sort table by {}"".format(sort)) + sort_column, sort_clause = parse_sort(sort) + if sort_column not in sortable_columns: + raise DatasetteError(""Cannot sort table by {}"".format(sort_column)) - order_by = escape_sqlite(sort) + order_by = sort_clause if sort_desc: - if sort_desc not in sortable_columns: - raise DatasetteError(""Cannot sort table by {}"".format(sort_desc)) + sort_column, sort_clause = parse_sort(sort_desc) + if sort_column not in sortable_columns: + raise DatasetteError(""Cannot sort table by {}"".format(sort_column)) - order_by = ""{} desc"".format(escape_sqlite(sort_desc)) + order_by = ""{} desc"".format(sort_clause) from_sql = ""from {table_name} {where}"".format( table_name=escape_sqlite(table), ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",657572753,?sort=colname~numeric to sort by by column cast to real, https://github.com/simonw/datasette/issues/894#issuecomment-709572425,https://api.github.com/repos/simonw/datasette/issues/894,709572425,MDEyOklzc3VlQ29tbWVudDcwOTU3MjQyNQ==,9599,simonw,2020-10-15T20:28:18Z,2020-10-15T20:28:18Z,OWNER,"Also need to rethink this template logic that decides if to show a column as sorted or not: https://github.com/simonw/datasette/blob/4f7c0ebd85ccd8c1853d7aa0147628f7c1b749cc/datasette/templates/_table.html#L10-L14","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",657572753,?sort=colname~numeric to sort by by column cast to real, https://github.com/simonw/datasette/issues/894#issuecomment-709571143,https://api.github.com/repos/simonw/datasette/issues/894,709571143,MDEyOklzc3VlQ29tbWVudDcwOTU3MTE0Mw==,9599,simonw,2020-10-15T20:25:35Z,2020-10-15T20:25:35Z,OWNER,"`cast(nullif(colname, '') as real)` can fix this - it will treat `''` the same as `null`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",657572753,?sort=colname~numeric to sort by by column cast to real, https://github.com/simonw/datasette/issues/894#issuecomment-709569951,https://api.github.com/repos/simonw/datasette/issues/894,709569951,MDEyOklzc3VlQ29tbWVudDcwOTU2OTk1MQ==,9599,simonw,2020-10-15T20:23:02Z,2020-10-15T20:23:02Z,OWNER,"Something to watch out for: `""""` empty strings cast to `0.0`: `select cast(""100"" as real), ""100"", cast(null as real), cast("""" as real)` cast(""100"" as real) | ""100"" | cast(null as real) | cast("""" as real) -- | -- | -- | -- 100.0 | 100 |   | 0.0 https://latest.datasette.io/fixtures?sql=select+cast%28%22100%22+as+real%29%2C+%22100%22%2C+cast%28null+as+real%29%2C+cast%28%22%22+as+real%29","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",657572753,?sort=colname~numeric to sort by by column cast to real, https://github.com/simonw/datasette/issues/894#issuecomment-709562940,https://api.github.com/repos/simonw/datasette/issues/894,709562940,MDEyOklzc3VlQ29tbWVudDcwOTU2Mjk0MA==,9599,simonw,2020-10-15T20:08:16Z,2020-10-15T20:08:16Z,OWNER,Relevant code: https://github.com/simonw/datasette/blob/4f7c0ebd85ccd8c1853d7aa0147628f7c1b749cc/datasette/views/table.py#L485-L510,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",657572753,?sort=colname~numeric to sort by by column cast to real, https://github.com/simonw/datasette/issues/894#issuecomment-709546976,https://api.github.com/repos/simonw/datasette/issues/894,709546976,MDEyOklzc3VlQ29tbWVudDcwOTU0Njk3Ng==,9599,simonw,2020-10-15T19:35:55Z,2020-10-15T19:36:38Z,OWNER,"Much easier solution: if the suffix is `~numeric` then treat it as the column name sorted numerically. If the suffix is missing OR the suffix is `~default`, sort without casting. Only add the `~default` suffix if the column name itself contains at least one `~` symbol. Using `~` because it doesn't need to be URL-encoded.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",657572753,?sort=colname~numeric to sort by by column cast to real, https://github.com/simonw/datasette/issues/894#issuecomment-709539257,https://api.github.com/repos/simonw/datasette/issues/894,709539257,MDEyOklzc3VlQ29tbWVudDcwOTUzOTI1Nw==,9599,simonw,2020-10-15T19:19:29Z,2020-10-15T19:34:07Z,OWNER,"Urgh this isn't going to work. `%7E~%7E` gets decoded as `~~~` so I wouldn't be able to tell the difference. I could use double-percentage-encoding here instead. I feel like there's a simpler solution that I'm missing (and that may well be in use within Datasette already, I'm not doing great thinking this morning).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",657572753,?sort=colname~numeric to sort by by column cast to real, https://github.com/simonw/datasette/issues/894#issuecomment-709534197,https://api.github.com/repos/simonw/datasette/issues/894,709534197,MDEyOklzc3VlQ29tbWVudDcwOTUzNDE5Nw==,9599,simonw,2020-10-15T19:08:53Z,2020-10-15T19:17:55Z,OWNER,"Even better solution: use URL encoding in the parameter details. This is consistent with how `?_next=` tokens work, e.g. `?_next=0.291861560261786%2Ce%2Cj`. So the format can be: - `mycolumn` - `urlencoded-mycolumn$castname` For most columns this will look like: `?_sort=score$numeric` For columns with a `$` in their name it will be `?_sort=score%24hasdollar$numeric` Problem: both `$` and `,` are usually URL encoded anyway. I need a character which isn't encoded by default, so that I can use its encoded form to show it is part of the column name and its un-encoded form to split the cast indicator. `_` is a candidate here - not encoded by default, but can be encoded as `%5F`. The other unreserved non-alphanumeric characters are `-`, `.`, `_`, `~`. Of these, `~` is least likely to show up in a column name. So I'll use that. - `mycolumn` - `mycolumn~numeric` - `mycolumn%7Ewith%7Etildes~numeric`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",657572753,?sort=colname~numeric to sort by by column cast to real, https://github.com/simonw/datasette/issues/894#issuecomment-709532369,https://api.github.com/repos/simonw/datasette/issues/894,709532369,MDEyOklzc3VlQ29tbWVudDcwOTUzMjM2OQ==,9599,simonw,2020-10-15T19:05:07Z,2020-10-15T19:07:35Z,OWNER,"Simpler option: `?_sort=` column values look like this: - `mycolumn` - for sort by column - `mycolumn$numeric` - for sort by column after cast to float - `mycolumn$morename$default` - for the edge case where the column name itself contains a $ symbol","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",657572753,?sort=colname~numeric to sort by by column cast to real, https://github.com/simonw/datasette/issues/894#issuecomment-709531343,https://api.github.com/repos/simonw/datasette/issues/894,709531343,MDEyOklzc3VlQ29tbWVudDcwOTUzMTM0Mw==,9599,simonw,2020-10-15T19:03:12Z,2020-10-15T19:03:12Z,OWNER,"The Sort by `
` which, when clicked, causes the dropdown menu to appear as an absolutely positioned `
` that is not located within the DOM hierarchy of the`
` itself but is positioned to show up in the correct place.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",711627628,Action menu for table columns, https://github.com/simonw/datasette/issues/981#issuecomment-701153822,https://api.github.com/repos/simonw/datasette/issues/981,701153822,MDEyOklzc3VlQ29tbWVudDcwMTE1MzgyMg==,9599,simonw,2020-09-30T04:47:10Z,2020-09-30T04:47:10Z,OWNER,"Future version could have expanding out nested side menus that let you do things like ""calculate sum/avg for this column against this-other-column"".","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",711627628,Action menu for table columns, https://github.com/simonw/datasette/issues/981#issuecomment-701153600,https://api.github.com/repos/simonw/datasette/issues/981,701153600,MDEyOklzc3VlQ29tbWVudDcwMTE1MzYwMA==,9599,simonw,2020-09-30T04:46:18Z,2020-09-30T04:46:18Z,OWNER,"More options: ```html ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",711627628,Action menu for table columns, https://github.com/simonw/datasette/issues/980#issuecomment-700929721,https://api.github.com/repos/simonw/datasette/issues/980,700929721,MDEyOklzc3VlQ29tbWVudDcwMDkyOTcyMQ==,9599,simonw,2020-09-29T19:21:50Z,2020-09-29T19:21:50Z,OWNER,"That fixed it: https://latest-with-plugins.datasette.io/fixtures?sql=select%0D%0A++dateutil_rrule(%27FREQ%3DHOURLY%3BCOUNT%3D5%27)%2C%0D%0A++dateutil_rrule_date(%0D%0A++++%27FREQ%3DDAILY%3BCOUNT%3D3%27%2C%0D%0A++++%271st+jan+2020%27%0D%0A++)%3B ```html ``` ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",710819020,Another rendering glitch with column headers on mobile, https://github.com/simonw/datasette/issues/980#issuecomment-700490225,https://api.github.com/repos/simonw/datasette/issues/980,700490225,MDEyOklzc3VlQ29tbWVudDcwMDQ5MDIyNQ==,9599,simonw,2020-09-29T06:53:37Z,2020-09-29T06:53:37Z,OWNER,"This time it's because there are newlines in the column header: ```html ``` Those need to be escaped somehow.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",710819020,Another rendering glitch with column headers on mobile, https://github.com/simonw/datasette/issues/979#issuecomment-700343373,https://api.github.com/repos/simonw/datasette/issues/979,700343373,MDEyOklzc3VlQ29tbWVudDcwMDM0MzM3Mw==,9599,simonw,2020-09-28T23:56:27Z,2020-09-28T23:56:27Z,OWNER,This would benefit https://github.com/simonw/datasette-import-table - which currently ignores the `CREATE TABLE` and derives the schema by inserting rows.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",710650633,Default table view JSON should include CREATE TABLE, https://github.com/simonw/datasette/issues/979#issuecomment-700343229,https://api.github.com/repos/simonw/datasette/issues/979,700343229,MDEyOklzc3VlQ29tbWVudDcwMDM0MzIyOQ==,9599,simonw,2020-09-28T23:55:55Z,2020-09-28T23:55:55Z,OWNER,Here's the code that adds it to the HTML context: https://github.com/simonw/datasette/blob/c11383e6284e000b2641569457efa16ac9e0d6ae/datasette/views/table.py#L835-L837,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",710650633,Default table view JSON should include CREATE TABLE, https://github.com/simonw/datasette/issues/978#issuecomment-700320480,https://api.github.com/repos/simonw/datasette/issues/978,700320480,MDEyOklzc3VlQ29tbWVudDcwMDMyMDQ4MA==,9599,simonw,2020-09-28T22:39:18Z,2020-09-28T22:39:18Z,OWNER,"```python def escape_css_string(s): return _css_re.sub(lambda m: ""\\"" + (""{:X}"".format(ord(m.group())).zfill(6)), s) ``` That fixes it: ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",710506708,Rendering glitch with column headings on mobile, https://github.com/simonw/datasette/issues/978#issuecomment-700319656,https://api.github.com/repos/simonw/datasette/issues/978,700319656,MDEyOklzc3VlQ29tbWVudDcwMDMxOTY1Ng==,9599,simonw,2020-09-28T22:36:44Z,2020-09-28T22:36:44Z,OWNER,"Weirdly even those leading 0s doesn't fix it: But... padding to six characters does! See https://www.w3.org/International/questions/qa-escapes ``` In [32]: print('\\' + ""{:X}"".format(ord('""')).zfill(6)) \000022 ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",710506708,Rendering glitch with column headings on mobile, https://github.com/simonw/datasette/issues/978#issuecomment-700317760,https://api.github.com/repos/simonw/datasette/issues/978,700317760,MDEyOklzc3VlQ29tbWVudDcwMDMxNzc2MA==,9599,simonw,2020-09-28T22:30:25Z,2020-09-28T22:30:25Z,OWNER,"```python print('\\' + ""{:X}"".format(ord('""')).zfill(4)) \0022 ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",710506708,Rendering glitch with column headings on mobile, https://github.com/simonw/datasette/issues/978#issuecomment-700316511,https://api.github.com/repos/simonw/datasette/issues/978,700316511,MDEyOklzc3VlQ29tbWVudDcwMDMxNjUxMQ==,9599,simonw,2020-09-28T22:26:38Z,2020-09-28T22:26:38Z,OWNER,The fix may be to use `\0022` instead of `\22`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",710506708,Rendering glitch with column headings on mobile, https://github.com/simonw/datasette/issues/978#issuecomment-700314509,https://api.github.com/repos/simonw/datasette/issues/978,700314509,MDEyOklzc3VlQ29tbWVudDcwMDMxNDUwOQ==,9599,simonw,2020-09-28T22:20:51Z,2020-09-28T22:20:51Z,OWNER,"Here's the HTML for the broken example above: ```html ``` The glitch affects the ones where the quote is followed by digits.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",710506708,Rendering glitch with column headings on mobile, https://github.com/simonw/datasette/issues/978#issuecomment-700313836,https://api.github.com/repos/simonw/datasette/issues/978,700313836,MDEyOklzc3VlQ29tbWVudDcwMDMxMzgzNg==,9599,simonw,2020-09-28T22:19:05Z,2020-09-28T22:19:05Z,OWNER,Looks like a bug in this function: https://github.com/simonw/datasette/blob/1f021c37110fc9019b0ef70062c28c335e568ae2/datasette/utils/__init__.py#L269-L274,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",710506708,Rendering glitch with column headings on mobile, https://github.com/simonw/datasette/pull/977#issuecomment-700012161,https://api.github.com/repos/simonw/datasette/issues/977,700012161,MDEyOklzc3VlQ29tbWVudDcwMDAxMjE2MQ==,22429695,codecov[bot],2020-09-28T13:37:44Z,2020-09-28T13:37:44Z,NONE,"# [Codecov](https://codecov.io/gh/simonw/datasette/pull/977?src=pr&el=h1) Report > Merging [#977](https://codecov.io/gh/simonw/datasette/pull/977?src=pr&el=desc) into [main](https://codecov.io/gh/simonw/datasette/commit/9a6d0dce282e7fb58c5610e24c74098c923abfdc?el=desc) will **not change** coverage. > The diff coverage is `n/a`. [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/977/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1)](https://codecov.io/gh/simonw/datasette/pull/977?src=pr&el=tree) ```diff @@ Coverage Diff @@ ## main #977 +/- ## ======================================= Coverage 84.27% 84.27% ======================================= Files 28 28 Lines 3847 3847 ======================================= Hits 3242 3242 Misses 605 605 ``` ------ [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/977?src=pr&el=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/977?src=pr&el=footer). Last update [9a6d0dc...5c01344](https://codecov.io/gh/simonw/datasette/pull/977?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",710269200,"Update pytest requirement from <6.1.0,>=5.2.2 to >=5.2.2,<6.2.0", https://github.com/simonw/sqlite-utils/issues/181#issuecomment-699762881,https://api.github.com/repos/simonw/sqlite-utils/issues/181,699762881,MDEyOklzc3VlQ29tbWVudDY5OTc2Mjg4MQ==,9599,simonw,2020-09-28T04:29:23Z,2020-09-28T04:29:23Z,OWNER,Relevant code: https://github.com/simonw/sqlite-utils/blob/94fc62857ee2655a21d85f6dae84b67bbfa5956d/sqlite_utils/db.py#L331-L367,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",709920027,"pk=[""id""] should have same effect as pk=""id""", https://github.com/simonw/sqlite-utils/issues/180#issuecomment-699718788,https://api.github.com/repos/simonw/sqlite-utils/issues/180,699718788,MDEyOklzc3VlQ29tbWVudDY5OTcxODc4OA==,9599,simonw,2020-09-28T01:11:45Z,2020-09-28T01:11:45Z,OWNER,https://hypothesis.readthedocs.io/,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",709861194,Try running some tests using Hypothesis, https://github.com/simonw/datasette/issues/858#issuecomment-699690034,https://api.github.com/repos/simonw/datasette/issues/858,699690034,MDEyOklzc3VlQ29tbWVudDY5OTY5MDAzNA==,39445562,smithdc1,2020-09-27T21:23:04Z,2020-09-27T21:23:04Z,NONE,"Hi Simon, Thanks so much for all your work on datasette, it's an excellent project and I wish you all the best with it. I particularly enjoyed your talk at the Django London Meetup a short while back. I've been trying to publish to Heroku from Windows 10 and I was running into this error. I'm not sure why it can't be run without `shell=True` on Windows but this seems to help. With this change, I am able to publish if I pass in a `name` to the `publish` command. When a `name` is not passed the default of `datasette` is used and therefore this line here fails (as datasette at heroku already exists) and causes the recession error mentioned above. https://github.com/simonw/datasette/blob/9a6d0dce282e7fb58c5610e24c74098c923abfdc/datasette/publish/heroku.py#L126 I tried to write a patch for this but I am really struggling with being on Windows (many of the tests seem to fail anyway?), and my lack of knowledge of Mock, so sorry for this. Hope this is of some help. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",642388564,publish heroku does not work on Windows 10, https://github.com/simonw/sqlite-utils/issues/179#issuecomment-699524671,https://api.github.com/repos/simonw/sqlite-utils/issues/179,699524671,MDEyOklzc3VlQ29tbWVudDY5OTUyNDY3MQ==,9599,simonw,2020-09-26T17:31:23Z,2020-09-27T20:31:50Z,OWNER,"SQL query for detecting integers: ```sql select 'contains_non_integer' as result from mytable where cast(cast(mycolumn AS INTEGER) AS TEXT) != mycolumn limit 1 ``` This will return a single row with a 1 as soon as it comes across a column that contains a non-integer - so it short circuits quickly on TEXT columns with non-integers in them. If everything in the column is an integer it will scan the whole thing before returning no rows. More extensive demo: ```sql select value, cast(cast(value AS INTEGER) AS TEXT) = value as is_valid_int from ( select '1' as value union select '1.1' as value union select 'dog' as value union select null as value ) ``` https://latest.datasette.io/fixtures?sql=select%0D%0A++value%2C%0D%0A++cast%28cast%28value+AS+INTEGER%29+AS+TEXT%29+%3D+value+as+is_valid_int%0D%0Afrom%0D%0A++%28%0D%0A++++select%0D%0A++++++%271%27+as+value%0D%0A++++union%0D%0A++++select%0D%0A++++++%271.1%27+as+value%0D%0A++++union%0D%0A++++select%0D%0A++++++%27dog%27+as+value%0D%0A++++union%0D%0A++++select%0D%0A++++++null+as+value%0D%0A++%29 value | is_valid_int -- | --   |   1 | 1 1.1 | 0 dog | 0","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",709577625,sqlite-utils transform/insert --detect-types, https://github.com/simonw/sqlite-utils/issues/179#issuecomment-699684535,https://api.github.com/repos/simonw/sqlite-utils/issues/179,699684535,MDEyOklzc3VlQ29tbWVudDY5OTY4NDUzNQ==,9599,simonw,2020-09-27T20:30:31Z,2020-09-27T20:30:31Z,OWNER,"This recipe looks like it might be the way to detect floats: ```sql select value, cast(cast(value AS REAL) AS TEXT) in (value, value || '.0') as is_valid_float from ( select '1' as value union select '1.1' as value union select 'dog' as value union select null as value ) ``` Demo: https://latest.datasette.io/fixtures?sql=select%0D%0A++value%2C%0D%0A++cast%28cast%28value+AS+REAL%29+AS+TEXT%29+in+%28value%2C+value+%7C%7C+%27.0%27%29+as+is_valid_float%0D%0Afrom%0D%0A++%28%0D%0A++++select%0D%0A++++++%271%27+as+value%0D%0A++++union%0D%0A++++select%0D%0A++++++%271.1%27+as+value%0D%0A++++union%0D%0A++++select%0D%0A++++++%27dog%27+as+value%0D%0A++++union%0D%0A++++select%0D%0A++++++null+as+value%0D%0A++%29 value | is_valid_float -- | --   |   1 | 1 1.1 | 1 dog | 0","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",709577625,sqlite-utils transform/insert --detect-types, https://github.com/simonw/sqlite-utils/issues/179#issuecomment-699526149,https://api.github.com/repos/simonw/sqlite-utils/issues/179,699526149,MDEyOklzc3VlQ29tbWVudDY5OTUyNjE0OQ==,9599,simonw,2020-09-26T17:43:28Z,2020-09-26T17:43:28Z,OWNER,Posed a question about this on the SQLite forum here: https://sqlite.org/forum/forumpost/ab0dcd66ef,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",709577625,sqlite-utils transform/insert --detect-types, https://github.com/simonw/sqlite-utils/issues/138#issuecomment-698626768,https://api.github.com/repos/simonw/sqlite-utils/issues/138,698626768,MDEyOklzc3VlQ29tbWVudDY5ODYyNjc2OA==,9599,simonw,2020-09-24T22:46:56Z,2020-09-24T22:46:56Z,OWNER,"Yeah this works fine, added a new confirmatory test.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",684118950,extracts= doesn't configure foreign keys, https://github.com/simonw/sqlite-utils/issues/173#issuecomment-698578959,https://api.github.com/repos/simonw/sqlite-utils/issues/173,698578959,MDEyOklzc3VlQ29tbWVudDY5ODU3ODk1OQ==,9599,simonw,2020-09-24T20:44:35Z,2020-09-24T20:50:19Z,OWNER,"I'm using a `click.File()` at the moment: https://github.com/simonw/sqlite-utils/blob/5a63b9e88c5887432eb1d7df39f304ea55038437/sqlite_utils/cli.py#L496 I'll need to change that to be something that I can easily measure progress through. Also I should change its name - `json_file` is a bad name when it sometimes handles `csv` or `tsv` instead. It looks like the argument provided by `click.File` doesn't provide a way to read the size of the file, so I need to switch that out for a file path instead. https://click.palletsprojects.com/en/7.x/api/#click.Path","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",707478649,Progress bar for sqlite-utils insert, https://github.com/simonw/sqlite-utils/issues/173#issuecomment-698579389,https://api.github.com/repos/simonw/sqlite-utils/issues/173,698579389,MDEyOklzc3VlQ29tbWVudDY5ODU3OTM4OQ==,9599,simonw,2020-09-24T20:45:29Z,2020-09-24T20:45:29Z,OWNER,"Relevant code: https://github.com/simonw/sqlite-utils/blob/5a63b9e88c5887432eb1d7df39f304ea55038437/sqlite_utils/cli.py#L550-L560 Changing that to track progress through NL-JSON, CSV and TSV shouldn't be too hard.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",707478649,Progress bar for sqlite-utils insert, https://github.com/simonw/sqlite-utils/issues/173#issuecomment-698577508,https://api.github.com/repos/simonw/sqlite-utils/issues/173,698577508,MDEyOklzc3VlQ29tbWVudDY5ODU3NzUwOA==,9599,simonw,2020-09-24T20:41:18Z,2020-09-24T20:41:18Z,OWNER,"I know how to build this for CSV and TSV - I can read them via a file wrapper that counts how many bytes it has seen. Not sure how to do it for JSON though. Maybe I could provide it just for newline-delimited JSON? Again I can measure progress based on how many bytes have been read.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",707478649,Progress bar for sqlite-utils insert, https://github.com/simonw/sqlite-utils/issues/119#issuecomment-698575545,https://api.github.com/repos/simonw/sqlite-utils/issues/119,698575545,MDEyOklzc3VlQ29tbWVudDY5ODU3NTU0NQ==,9599,simonw,2020-09-24T20:36:59Z,2020-09-24T20:36:59Z,OWNER,This was implemented in #161.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",652700770,Ability to remove a foreign key, https://github.com/simonw/sqlite-utils/issues/176#issuecomment-698572493,https://api.github.com/repos/simonw/sqlite-utils/issues/176,698572493,MDEyOklzc3VlQ29tbWVudDY5ODU3MjQ5Mw==,9599,simonw,2020-09-24T20:30:18Z,2020-09-24T20:30:18Z,OWNER,Documentation: https://sqlite-utils.readthedocs.io/en/stable/cli.html#transforming-tables,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",708293114,sqlite-utils transform column order option, https://github.com/simonw/sqlite-utils/issues/175#issuecomment-698572264,https://api.github.com/repos/simonw/sqlite-utils/issues/175,698572264,MDEyOklzc3VlQ29tbWVudDY5ODU3MjI2NA==,9599,simonw,2020-09-24T20:29:48Z,2020-09-24T20:29:48Z,OWNER,Documentation: https://sqlite-utils.readthedocs.io/en/stable/python-api.html#transforming-a-table,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",708261775,Add docs for .transform(column_order=), https://github.com/simonw/datasette/issues/976#issuecomment-698488971,https://api.github.com/repos/simonw/datasette/issues/976,698488971,MDEyOklzc3VlQ29tbWVudDY5ODQ4ODk3MQ==,9599,simonw,2020-09-24T17:42:09Z,2020-09-24T17:42:35Z,OWNER,This is complex enough new logic that it will need test coverage - specifically covering tables or databases with strange names.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",708289783,Idea: -o could open to a more convenient location, https://github.com/simonw/sqlite-utils/issues/177#issuecomment-698444567,https://api.github.com/repos/simonw/sqlite-utils/issues/177,698444567,MDEyOklzc3VlQ29tbWVudDY5ODQ0NDU2Nw==,9599,simonw,2020-09-24T16:14:47Z,2020-09-24T16:14:47Z,OWNER,"This is a backwards incompatible change, so technically I should bump the major version to 3. I'm not going to do that, because the feature is brand new and the chance that anyone has written code or shell scripts that use it is vanishingly small.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",708301810,Simplify .transform(drop_foreign_keys=) and sqlite-transform --drop-foreign-key, https://github.com/simonw/sqlite-utils/issues/176#issuecomment-698438043,https://api.github.com/repos/simonw/sqlite-utils/issues/176,698438043,MDEyOklzc3VlQ29tbWVudDY5ODQzODA0Mw==,9599,simonw,2020-09-24T16:02:55Z,2020-09-24T16:02:55Z,OWNER,I think I'll call this option `--column-order` with a shortcut of `-o`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",708293114,sqlite-utils transform column order option, https://github.com/simonw/sqlite-utils/issues/175#issuecomment-698434811,https://api.github.com/repos/simonw/sqlite-utils/issues/175,698434811,MDEyOklzc3VlQ29tbWVudDY5ODQzNDgxMQ==,9599,simonw,2020-09-24T15:57:17Z,2020-09-24T15:57:17Z,OWNER,Landed that.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",708261775,Add docs for .transform(column_order=), https://github.com/simonw/datasette/issues/970#issuecomment-698434236,https://api.github.com/repos/simonw/datasette/issues/970,698434236,MDEyOklzc3VlQ29tbWVudDY5ODQzNDIzNg==,9599,simonw,2020-09-24T15:56:18Z,2020-09-24T15:56:50Z,OWNER,"Idea: if a database only has a single table, this could open straight to `/db/table`. If it has multiple tables but a single database it could open straight to `/db`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705108492,"request an ""-o"" option on ""datasette server"" to open the default browser at the running url", https://github.com/simonw/sqlite-utils/issues/175#issuecomment-698412692,https://api.github.com/repos/simonw/sqlite-utils/issues/175,698412692,MDEyOklzc3VlQ29tbWVudDY5ODQxMjY5Mg==,9599,simonw,2020-09-24T15:19:28Z,2020-09-24T15:19:28Z,OWNER,Need to land #174 first.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",708261775,Add docs for .transform(column_order=), https://github.com/simonw/sqlite-utils/pull/174#issuecomment-698400790,https://api.github.com/repos/simonw/sqlite-utils/issues/174,698400790,MDEyOklzc3VlQ29tbWVudDY5ODQwMDc5MA==,9599,simonw,2020-09-24T14:59:50Z,2020-09-24T14:59:50Z,OWNER,For reusing the lookup table: I'm going to raise an error if a lookup table exists but without the correct columns. The caller can then add those columns and try again.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",707944044,"Much, much faster extract() implementation", https://github.com/simonw/sqlite-utils/pull/174#issuecomment-698184166,https://api.github.com/repos/simonw/sqlite-utils/issues/174,698184166,MDEyOklzc3VlQ29tbWVudDY5ODE4NDE2Ng==,9599,simonw,2020-09-24T08:01:07Z,2020-09-24T08:01:07Z,OWNER,I may revert the now unnecessary undocumented tweaks to the `.update()` method made in 66d506587eba9f0715267d6560b97c1fa44cc781 as well.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",707944044,"Much, much faster extract() implementation", https://github.com/simonw/sqlite-utils/pull/174#issuecomment-698182656,https://api.github.com/repos/simonw/sqlite-utils/issues/174,698182656,MDEyOklzc3VlQ29tbWVudDY5ODE4MjY1Ng==,9599,simonw,2020-09-24T07:58:08Z,2020-09-24T07:58:08Z,OWNER,"The way the lookup table works here differs from the previous implementation. In the previous implementation the usage of `.lookup()` meant that an existing table would be modified to fit the new purpose. That no longer happens in this version. Need to make a design decision about how this should work. It should definitely be possible to use an existing lookup table - imagine a database where several tables have a ""Departments"" column and we want to extract all of those values out to a single shared ""Departments"" table.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",707944044,"Much, much faster extract() implementation", https://github.com/simonw/sqlite-utils/pull/174#issuecomment-698182037,https://api.github.com/repos/simonw/sqlite-utils/issues/174,698182037,MDEyOklzc3VlQ29tbWVudDY5ODE4MjAzNw==,9599,simonw,2020-09-24T07:56:50Z,2020-09-24T07:56:50Z,OWNER,I could also be a bit smarter about transaction handling. I think it may be possible to run this entire operation in a single transaction now.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",707944044,"Much, much faster extract() implementation", https://github.com/simonw/sqlite-utils/pull/174#issuecomment-698181478,https://api.github.com/repos/simonw/sqlite-utils/issues/174,698181478,MDEyOklzc3VlQ29tbWVudDY5ODE4MTQ3OA==,9599,simonw,2020-09-24T07:55:45Z,2020-09-24T07:55:45Z,OWNER,`import functools` is no longer needed.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",707944044,"Much, much faster extract() implementation", https://github.com/simonw/sqlite-utils/pull/174#issuecomment-698180705,https://api.github.com/repos/simonw/sqlite-utils/issues/174,698180705,MDEyOklzc3VlQ29tbWVudDY5ODE4MDcwNQ==,9599,simonw,2020-09-24T07:54:10Z,2020-09-24T07:54:10Z,OWNER,"After running through the steps in https://simonwillison.net/2020/Sep/23/sqlite-utils-extract/ I get a table that looks like this: The foreign key columns are all at the end of the table. It would be nicer if they were arranged in the same order as the columns they replaced.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",707944044,"Much, much faster extract() implementation", https://github.com/simonw/sqlite-utils/pull/174#issuecomment-698180113,https://api.github.com/repos/simonw/sqlite-utils/issues/174,698180113,MDEyOklzc3VlQ29tbWVudDY5ODE4MDExMw==,9599,simonw,2020-09-24T07:53:03Z,2020-09-24T07:53:03Z,OWNER,This could do with a little bit more testing - I'm worried there may be column or table name edge cases that are not covered yet. I also need to remove the progress bar code since that no longer makes sense for this implementation.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",707944044,"Much, much faster extract() implementation", https://github.com/simonw/sqlite-utils/issues/172#issuecomment-698178101,https://api.github.com/repos/simonw/sqlite-utils/issues/172,698178101,MDEyOklzc3VlQ29tbWVudDY5ODE3ODEwMQ==,9599,simonw,2020-09-24T07:48:57Z,2020-09-24T07:49:20Z,OWNER,"> I wonder if I could make this faster by separating it out into a few steps: > > * Create the new lookup table with all of the distinct rows > > * Add the blank foreign key column > > * run a `UPDATE table SET blah_id = (select id from lookup where thang = table.thang)` > > * Drop the value columns My prototype of this knocked the time down from 10 minutes to 4 seconds, so I think the change is worth it! ``` % date sqlite-utils extract salaries.db salaries \ 'Department Code' 'Department' \ --table 'departments' \ --fk-column 'department_id' \ --rename 'Department Code' code \ --rename 'Department' name date sqlite-utils extract salaries.db salaries \ 'Union Code' 'Union' \ --table 'unions' \ --fk-column 'union_id' \ --rename 'Union Code' code \ --rename 'Union' name date sqlite-utils extract salaries.db salaries \ 'Job Family Code' 'Job Family' \ --table 'job_families' \ --fk-column 'job_family_id' \ --rename 'Job Family Code' code \ --rename 'Job Family' name date sqlite-utils extract salaries.db salaries \ 'Job Code' 'Job' \ --table 'jobs' \ --fk-column 'job_id' \ --rename 'Job Code' code \ --rename 'Job' name date Thu Sep 24 00:48:16 PDT 2020 Thu Sep 24 00:48:20 PDT 2020 Thu Sep 24 00:48:24 PDT 2020 Thu Sep 24 00:48:28 PDT 2020 Thu Sep 24 00:48:32 PDT 2020 ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",707427200,Improve performance of extract operations, https://github.com/simonw/datasette/issues/123#issuecomment-698174957,https://api.github.com/repos/simonw/datasette/issues/123,698174957,MDEyOklzc3VlQ29tbWVudDY5ODE3NDk1Nw==,45416,obra,2020-09-24T07:42:05Z,2020-09-24T07:42:05Z,NONE," Oh. Awesome. On Thu, Sep 24, 2020 at 12:28:53AM -0700, Simon Willison wrote: > @obra there's a plugin for that! https://github.com/simonw/ > datasette-upload-csvs > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub, or unsubscribe.* > -- ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",275125561,Datasette serve should accept paths/URLs to CSVs and other file formats, https://github.com/simonw/datasette/issues/123#issuecomment-698168648,https://api.github.com/repos/simonw/datasette/issues/123,698168648,MDEyOklzc3VlQ29tbWVudDY5ODE2ODY0OA==,9599,simonw,2020-09-24T07:28:38Z,2020-09-24T07:28:38Z,OWNER,@obra there's a plugin for that! https://github.com/simonw/datasette-upload-csvs,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",275125561,Datasette serve should accept paths/URLs to CSVs and other file formats, https://github.com/simonw/datasette/issues/974#issuecomment-698110492,https://api.github.com/repos/simonw/datasette/issues/974,698110492,MDEyOklzc3VlQ29tbWVudDY5ODExMDQ5Mg==,9599,simonw,2020-09-24T04:50:56Z,2020-09-24T04:51:05Z,OWNER,"Come to think of it I've noticed that in the logs when it's running on my laptop, definitely worth fixing.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",707849175,static assets and favicon aren't cached by the browser, https://github.com/simonw/datasette/issues/123#issuecomment-698110186,https://api.github.com/repos/simonw/datasette/issues/123,698110186,MDEyOklzc3VlQ29tbWVudDY5ODExMDE4Ng==,45416,obra,2020-09-24T04:49:51Z,2020-09-24T04:49:51Z,NONE,"As a half-measure, I'd get value out of being able to upload a CSV and have datasette run csv-to-sqlite on it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",275125561,Datasette serve should accept paths/URLs to CSVs and other file formats, https://github.com/simonw/datasette/issues/619#issuecomment-698024773,https://api.github.com/repos/simonw/datasette/issues/619,698024773,MDEyOklzc3VlQ29tbWVudDY5ODAyNDc3Mw==,9599,simonw,2020-09-23T23:31:46Z,2020-09-23T23:31:46Z,OWNER,"I'm going to have to untangle Datasette's error handling a bit for this - currently the expectation is that exceptions will be handled at a higher level, but I need to rethink that to make it cleaner for views like the ""execute custom SQL"" view to add their own error handling (and still be able to return the correct HTTP status codes, even with custom pages).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520655983,"""Invalid SQL"" page should let you edit the SQL", https://github.com/simonw/datasette/issues/619#issuecomment-697998045,https://api.github.com/repos/simonw/datasette/issues/619,697998045,MDEyOklzc3VlQ29tbWVudDY5Nzk5ODA0NQ==,9599,simonw,2020-09-23T22:09:06Z,2020-09-23T22:09:06Z,OWNER,"I'll add this to the succesful JSON format: ```json { ""ok"": true, ""error"": null } ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520655983,"""Invalid SQL"" page should let you edit the SQL", https://github.com/simonw/datasette/issues/619#issuecomment-697995885,https://api.github.com/repos/simonw/datasette/issues/619,697995885,MDEyOklzc3VlQ29tbWVudDY5Nzk5NTg4NQ==,9599,simonw,2020-09-23T22:02:44Z,2020-09-23T22:08:28Z,OWNER,"So the JSON (still served with a 500 code) will look something like this: ```json { ""ok"": false, ""status"": 500, ""database"": ""fixtures"", ""query_name"": null, ""rows"": [], ""truncated"": false, ""error"": ""Error message goes here"", ""columns"": [], ""query"": { ""sql"": ""the query that broke goes here"", ""params"": {} }, ""private"": false, ""allow_execute_sql"": true, ""query_ms"": 0.8716583251953125, ""source"": ""tests/fixtures.py"", ""source_url"": ""https://github.com/simonw/datasette/blob/master/tests/fixtures.py"", ""license"": ""Apache License 2.0"", ""license_url"": ""https://github.com/simonw/datasette/blob/master/LICENSE"" } ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520655983,"""Invalid SQL"" page should let you edit the SQL", https://github.com/simonw/datasette/issues/619#issuecomment-697995303,https://api.github.com/repos/simonw/datasette/issues/619,697995303,MDEyOklzc3VlQ29tbWVudDY5Nzk5NTMwMw==,9599,simonw,2020-09-23T22:01:08Z,2020-09-23T22:01:08Z,OWNER,"This is a little tricky to solve, because of the location of the form and the need to return JSON as well as HTML. It would be weird if a JSON request came in and got back the standard output from https://latest.datasette.io/fixtures.json when they were expecting to get back JSON in the shape of https://latest.datasette.io/fixtures.json?sql=select%20*%20from%20sqlite_master I'm going to return the HTML view that you would get for 0 results for a query - https://latest.datasette.io/fixtures?sql=select%201%20limit%200 - but with an error message.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520655983,"""Invalid SQL"" page should let you edit the SQL", https://github.com/simonw/datasette/issues/619#issuecomment-697980061,https://api.github.com/repos/simonw/datasette/issues/619,697980061,MDEyOklzc3VlQ29tbWVudDY5Nzk4MDA2MQ==,9599,simonw,2020-09-23T21:22:42Z,2020-09-23T21:22:42Z,OWNER,Yeah that sucks. Bumping this up the priority list.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520655983,"""Invalid SQL"" page should let you edit the SQL", https://github.com/simonw/datasette/issues/619#issuecomment-697973420,https://api.github.com/repos/simonw/datasette/issues/619,697973420,MDEyOklzc3VlQ29tbWVudDY5Nzk3MzQyMA==,45416,obra,2020-09-23T21:07:58Z,2020-09-23T21:07:58Z,NONE,"I've just run into this after crafting a complex query and discovered that hitting back loses my query. Even showing me the whole bad query would be a huge improvement over the current status quo.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",520655983,"""Invalid SQL"" page should let you edit the SQL", https://github.com/simonw/sqlite-utils/issues/172#issuecomment-697869886,https://api.github.com/repos/simonw/sqlite-utils/issues/172,697869886,MDEyOklzc3VlQ29tbWVudDY5Nzg2OTg4Ng==,9599,simonw,2020-09-23T18:45:30Z,2020-09-23T18:45:30Z,OWNER,"There's something to be said for making this operation pausable and resumable, especially if I'm going to make it available in a Datasette plugin at some point.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",707427200,Improve performance of extract operations, https://github.com/simonw/sqlite-utils/issues/172#issuecomment-697866885,https://api.github.com/repos/simonw/sqlite-utils/issues/172,697866885,MDEyOklzc3VlQ29tbWVudDY5Nzg2Njg4NQ==,9599,simonw,2020-09-23T18:43:37Z,2020-09-23T18:43:37Z,OWNER,Also what would happen if the table had new rows added to it while that command was running?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",707427200,Improve performance of extract operations, https://github.com/simonw/sqlite-utils/issues/172#issuecomment-697863116,https://api.github.com/repos/simonw/sqlite-utils/issues/172,697863116,MDEyOklzc3VlQ29tbWVudDY5Nzg2MzExNg==,9599,simonw,2020-09-23T18:41:06Z,2020-09-23T18:41:06Z,OWNER,Problem with this approach is it's not compatible with progress bars - but if it's a multiple of times faster it's worth it.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",707427200,Improve performance of extract operations, https://github.com/simonw/sqlite-utils/issues/172#issuecomment-697859772,https://api.github.com/repos/simonw/sqlite-utils/issues/172,697859772,MDEyOklzc3VlQ29tbWVudDY5Nzg1OTc3Mg==,9599,simonw,2020-09-23T18:38:43Z,2020-09-23T18:38:52Z,OWNER,"I wonder if I could make this faster by separating it out into a few steps: - Create the new lookup table with all of the distinct rows - Add the blank foreign key column - run a `UPDATE table SET blah_id = (select id from lookup where thang = table.thang)` - Drop the value columns","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",707427200,Improve performance of extract operations, https://github.com/simonw/sqlite-utils/issues/172#issuecomment-697835956,https://api.github.com/repos/simonw/sqlite-utils/issues/172,697835956,MDEyOklzc3VlQ29tbWVudDY5NzgzNTk1Ng==,9599,simonw,2020-09-23T18:22:49Z,2020-09-23T18:22:49Z,OWNER,"I ran `sudo py-spy top -p 123` against the process while it was running and the most time is definitely spent in `.update()`: ``` Total Samples 1000 GIL: 0.00%, Active: 90.00%, Threads: 1 %Own %Total OwnTime TotalTime Function (filename:line) 38.00% 38.00% 3.85s 3.85s update (sqlite_utils/db.py:1283) 27.00% 27.00% 2.12s 2.12s execute (sqlite_utils/db.py:161) 10.00% 10.00% 0.890s 0.890s execute (sqlite_utils/db.py:163) 10.00% 17.00% 0.870s 1.54s columns (sqlite_utils/db.py:553) 0.00% 0.00% 0.110s 0.210s (sqlite_utils/db.py:554) 0.00% 3.00% 0.100s 0.320s table_names (sqlite_utils/db.py:191) 0.00% 0.00% 0.100s 0.100s __new__ (:1) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",707427200,Improve performance of extract operations, https://github.com/simonw/sqlite-utils/issues/173#issuecomment-697577646,https://api.github.com/repos/simonw/sqlite-utils/issues/173,697577646,MDEyOklzc3VlQ29tbWVudDY5NzU3NzY0Ng==,9599,simonw,2020-09-23T15:48:51Z,2020-09-23T15:48:51Z,OWNER,"This can only work when it's reading from a file, not when it's reading from standard input.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",707478649,Progress bar for sqlite-utils insert, https://github.com/simonw/datasette/issues/111#issuecomment-697545290,https://api.github.com/repos/simonw/datasette/issues/111,697545290,MDEyOklzc3VlQ29tbWVudDY5NzU0NTI5MA==,9599,simonw,2020-09-23T15:29:11Z,2020-09-23T15:29:11Z,OWNER,This is still a good idea.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",274615452,Add “updated” to metadata, https://github.com/simonw/sqlite-utils/issues/172#issuecomment-697473247,https://api.github.com/repos/simonw/sqlite-utils/issues/172,697473247,MDEyOklzc3VlQ29tbWVudDY5NzQ3MzI0Nw==,9599,simonw,2020-09-23T14:45:13Z,2020-09-23T14:45:13Z,OWNER,"`lookup_table.lookup(lookups)` is doing a SQL lookup. This could be cached in-memory, maybe with a LRU cache, to avoid looking up the primary key for records that we have recently used. The `.update()` method it is calling first does a `get()` and then does a SQL `UPDATE ... WHERE`: https://github.com/simonw/sqlite-utils/blob/1ebffe1dbeaed7311e5b61ed988f4cd701e84808/sqlite_utils/db.py#L1244-L1264 Batching those updates may have an effect. Or finding a way to skip the `.get()` since we already know we have a valid record. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",707427200,Improve performance of extract operations, https://github.com/simonw/sqlite-utils/issues/172#issuecomment-697467833,https://api.github.com/repos/simonw/sqlite-utils/issues/172,697467833,MDEyOklzc3VlQ29tbWVudDY5NzQ2NzgzMw==,9599,simonw,2020-09-23T14:42:03Z,2020-09-23T14:42:03Z,OWNER,Here's the loop that's taking the time: https://github.com/simonw/sqlite-utils/blob/1ebffe1dbeaed7311e5b61ed988f4cd701e84808/sqlite_utils/db.py#L892-L897,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",707427200,Improve performance of extract operations, https://github.com/simonw/sqlite-utils/issues/172#issuecomment-697466497,https://api.github.com/repos/simonw/sqlite-utils/issues/172,697466497,MDEyOklzc3VlQ29tbWVudDY5NzQ2NjQ5Nw==,9599,simonw,2020-09-23T14:41:17Z,2020-09-23T14:41:17Z,OWNER,"Steps to produce that database: ``` curl -o salaries.csv 'https://data.sfgov.org/api/views/88g8-5mnd/rows.csv?accessType=DOWNLOAD' sqlite-utils insert salaries.db salaries salaries.csv --csv ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",707427200,Improve performance of extract operations, https://github.com/simonw/datasette/issues/970#issuecomment-697073465,https://api.github.com/repos/simonw/datasette/issues/970,697073465,MDEyOklzc3VlQ29tbWVudDY5NzA3MzQ2NQ==,2861690,secretGeek,2020-09-23T01:49:05Z,2020-09-23T01:49:05Z,NONE,"Oh wow oh wow. Thanks so much Simon. In an astoundingly rough week, this is a shining jewel. 🤣","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705108492,"request an ""-o"" option on ""datasette server"" to open the default browser at the running url", https://github.com/simonw/sqlite-utils/issues/170#issuecomment-697047591,https://api.github.com/repos/simonw/sqlite-utils/issues/170,697047591,MDEyOklzc3VlQ29tbWVudDY5NzA0NzU5MQ==,9599,simonw,2020-09-23T00:14:52Z,2020-09-23T00:14:52Z,OWNER," @simonw @db.register_function decorator, closes #162 4824775 @simonw table.transform() method - closes #114 987dd12 @simonw Keyword only arguments for transform() f8e10df Also renamed columns= to types= Closes #165 Commits on Sep 22, 2020 @simonw Implemented sqlite-utils transform command, closes #164 752d261 @simonw Applied Black f29f682 @simonw table.extract() method, refs #42 f855379 @simonw Docstring for sqlite-utils transform c755f28 @simonw Added table.extract(rename=) option, refs #42 c3210f2 @simonw Applied Black 317071a @simonw New .rows_where(select=) argument 7178231 @simonw table.extract() now works with rowid tables, refs #42 2db6c5b @simonw sqlite-utils extract, closes #42 55cf928 @simonw Progress bar for ""sqlite-utils extract"", closes #169 5c4d58d @simonw Fixed PRAGMA foreign_keys handling for .transform, closes #167 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",706768798,Release notes for 2.20, https://github.com/simonw/sqlite-utils/issues/42#issuecomment-697037974,https://api.github.com/repos/simonw/sqlite-utils/issues/42,697037974,MDEyOklzc3VlQ29tbWVudDY5NzAzNzk3NA==,9599,simonw,2020-09-22T23:39:31Z,2020-09-22T23:39:31Z,OWNER,Documentation for `sqlite-utils extract`: https://sqlite-utils.readthedocs.io/en/latest/cli.html#extracting-columns-into-a-separate-table,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",470345929,"table.extract(...) method and ""sqlite-utils extract"" command", https://github.com/simonw/sqlite-utils/issues/42#issuecomment-697031174,https://api.github.com/repos/simonw/sqlite-utils/issues/42,697031174,MDEyOklzc3VlQ29tbWVudDY5NzAzMTE3NA==,9599,simonw,2020-09-22T23:16:00Z,2020-09-22T23:16:00Z,OWNER,"Trying this demo again: ``` wget 'https://raw.githubusercontent.com/wri/global-power-plant-database/master/output_database/global_power_plant_database.csv' sqlite-utils insert global.db power_plants global_power_plant_database.csv --csv sqlite-utils extract global.db power_plants country country_long --table countries --rename country_long name ``` It worked!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",470345929,"table.extract(...) method and ""sqlite-utils extract"" command", https://github.com/simonw/sqlite-utils/issues/42#issuecomment-697025403,https://api.github.com/repos/simonw/sqlite-utils/issues/42,697025403,MDEyOklzc3VlQ29tbWVudDY5NzAyNTQwMw==,9599,simonw,2020-09-22T22:57:53Z,2020-09-22T22:57:53Z,OWNER,The documentation for the `.extract()` method is here: https://sqlite-utils.readthedocs.io/en/latest/python-api.html#extracting-columns-into-a-separate-table,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",470345929,"table.extract(...) method and ""sqlite-utils extract"" command", https://github.com/simonw/sqlite-utils/issues/42#issuecomment-697019944,https://api.github.com/repos/simonw/sqlite-utils/issues/42,697019944,MDEyOklzc3VlQ29tbWVudDY5NzAxOTk0NA==,9599,simonw,2020-09-22T22:40:00Z,2020-09-22T22:40:00Z,OWNER,"I tried out the prototype of the CLI on the Global Power Plants data: ``` wget 'https://raw.githubusercontent.com/wri/global-power-plant-database/master/output_database/global_power_plant_database.csv' sqlite-utils insert global.db power_plants global_power_plant_database.csv --csv sqlite-utils extract global.db power_plants country country_long ``` This threw an error because `rowid` columns are not yet supported. I fixed that like so: ``` sqlite-utils transform global.db power_plants --rename rowid id sqlite-utils extract global.db power_plants country country_long ``` That worked! But it didn't play great with Datasette, because the resulting extracted table had columns `country` and `country_long` and neither of those are called `name` or `value` or `title`. Based on this I need to add `rowid` table support AND I need to implement the proposed `rename=` argument for renaming columns on their way into the new table. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",470345929,"table.extract(...) method and ""sqlite-utils extract"" command", https://github.com/simonw/sqlite-utils/issues/42#issuecomment-697013681,https://api.github.com/repos/simonw/sqlite-utils/issues/42,697013681,MDEyOklzc3VlQ29tbWVudDY5NzAxMzY4MQ==,9599,simonw,2020-09-22T22:22:49Z,2020-09-22T22:22:49Z,OWNER,"The command-line version of this needs to accept a table and one or more columns, then a `--table` and `--fk-column` option.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",470345929,"table.extract(...) method and ""sqlite-utils extract"" command", https://github.com/simonw/sqlite-utils/issues/42#issuecomment-697012111,https://api.github.com/repos/simonw/sqlite-utils/issues/42,697012111,MDEyOklzc3VlQ29tbWVudDY5NzAxMjExMQ==,9599,simonw,2020-09-22T22:18:13Z,2020-09-22T22:18:13Z,OWNER,"Here's how I'm generating the examples for the documentation: ``` In [2]: import sqlite_utils In [3]: db = sqlite_utils.Database(memory=True) In [4]: db[""Trees""].insert({""id"": 1, ""TreeAddress"": ""52 Vine St"", ""CommonName"": ...: ""Palm"", ""LatinName"": ""foo""}, pk=""id"") Out[4]: In [5]: db[""Trees""].extract([""CommonName"", ""LatinName""], table=""Species"", fk_col ...: umn=""species_id"") In [6]: print(db[""Trees""].schema) CREATE TABLE ""Trees"" ( [id] INTEGER PRIMARY KEY, [TreeAddress] TEXT, [species_id] INTEGER, FOREIGN KEY(species_id) REFERENCES Species(id) ) In [7]: print(db[""Species""].schema) CREATE TABLE [Species] ( [id] INTEGER PRIMARY KEY, [CommonName] TEXT, [LatinName] TEXT ) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",470345929,"table.extract(...) method and ""sqlite-utils extract"" command", https://github.com/simonw/sqlite-utils/issues/42#issuecomment-696987925,https://api.github.com/repos/simonw/sqlite-utils/issues/42,696987925,MDEyOklzc3VlQ29tbWVudDY5Njk4NzkyNQ==,9599,simonw,2020-09-22T21:19:04Z,2020-09-22T21:19:04Z,OWNER,Need to make sure this works correctly for `rowid` tables.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",470345929,"table.extract(...) method and ""sqlite-utils extract"" command", https://github.com/simonw/sqlite-utils/issues/42#issuecomment-696987257,https://api.github.com/repos/simonw/sqlite-utils/issues/42,696987257,MDEyOklzc3VlQ29tbWVudDY5Njk4NzI1Nw==,9599,simonw,2020-09-22T21:17:34Z,2020-09-22T21:17:34Z,OWNER,"What to do if the table already exists? The `.lookup()` function already knows how to modify an existing table to create the correct constraints etc, so I'll rely on that mechanism.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",470345929,"table.extract(...) method and ""sqlite-utils extract"" command", https://github.com/simonw/sqlite-utils/issues/42#issuecomment-696980709,https://api.github.com/repos/simonw/sqlite-utils/issues/42,696980709,MDEyOklzc3VlQ29tbWVudDY5Njk4MDcwOQ==,9599,simonw,2020-09-22T21:05:07Z,2020-09-22T21:05:07Z,OWNER,"So `.extract()` probably takes a `batch_size=` argument too, which defaults to maybe 1000.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",470345929,"table.extract(...) method and ""sqlite-utils extract"" command", https://github.com/simonw/sqlite-utils/issues/42#issuecomment-696980503,https://api.github.com/repos/simonw/sqlite-utils/issues/42,696980503,MDEyOklzc3VlQ29tbWVudDY5Njk4MDUwMw==,9599,simonw,2020-09-22T21:04:45Z,2020-09-22T21:04:45Z,OWNER,"`table.extract()` can take an optional `progress=` argument which is a callback which will be used to report progress - called after each batch with `(num_done, total)`. It will get called with `(0, total)` once at the start to allow progress bars to be initialized. The command-line progress bar will use this.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",470345929,"table.extract(...) method and ""sqlite-utils extract"" command", https://github.com/simonw/sqlite-utils/issues/42#issuecomment-696979626,https://api.github.com/repos/simonw/sqlite-utils/issues/42,696979626,MDEyOklzc3VlQ29tbWVudDY5Njk3OTYyNg==,9599,simonw,2020-09-22T21:03:11Z,2020-09-22T21:03:11Z,OWNER,"And if you want to rename some of the columns in the new table: ```python db[""trees""].extract([""common_name"", ""latin_name""], table=""species"", rename={""common_name"": ""name""}) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",470345929,"table.extract(...) method and ""sqlite-utils extract"" command", https://github.com/simonw/sqlite-utils/issues/42#issuecomment-696979168,https://api.github.com/repos/simonw/sqlite-utils/issues/42,696979168,MDEyOklzc3VlQ29tbWVudDY5Njk3OTE2OA==,9599,simonw,2020-09-22T21:02:24Z,2020-09-22T21:02:24Z,OWNER,"In Python it looks like this: ```python # Simple case - species column species_id pointing to species table db[""trees""].extract(""species"") # Setting a custom table db[""trees""].extract(""species"", table=""Species"") # Custom foreign key column on trees db[""trees""].extract(""species"", fk_column=""species"") # Extracting multiple columns db[""trees""].extract([""common_name"", ""latin_name""]) # (this creates a lookup table called common_name_latin_name ref'd by common_name_latin_name_id) # Or with explicit table (fk_column here defaults to species_id because of the table name) db[""trees""].extract([""common_name"", ""latin_name""], table=""species"") ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",470345929,"table.extract(...) method and ""sqlite-utils extract"" command", https://github.com/simonw/sqlite-utils/issues/42#issuecomment-696976678,https://api.github.com/repos/simonw/sqlite-utils/issues/42,696976678,MDEyOklzc3VlQ29tbWVudDY5Njk3NjY3OA==,9599,simonw,2020-09-22T20:57:57Z,2020-09-22T20:57:57Z,OWNER,"I think I understand the shape of this feature now. It lets you specify one or more columns on the source table which will be extracted into another table. It uses the `.lookup()` mechanism to populate that other table, which means each unique column value / pair / triple will be assigned an integer ID. That integer ID gets written back into the first of the columns that are being transformed. A `.transform()` call then converts that column to an integer (and drops the additional columns). Finally we set up the new foreign key relationship.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",470345929,"table.extract(...) method and ""sqlite-utils extract"" command", https://github.com/simonw/sqlite-utils/issues/42#issuecomment-696893774,https://api.github.com/repos/simonw/sqlite-utils/issues/42,696893774,MDEyOklzc3VlQ29tbWVudDY5Njg5Mzc3NA==,9599,simonw,2020-09-22T18:15:33Z,2020-09-22T18:15:33Z,OWNER,I think the new foreign key column is called `company_name_id` by default in this example but can be customized by passing `--fk-column=xxx`,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",470345929,"table.extract(...) method and ""sqlite-utils extract"" command", https://github.com/simonw/sqlite-utils/issues/42#issuecomment-696893244,https://api.github.com/repos/simonw/sqlite-utils/issues/42,696893244,MDEyOklzc3VlQ29tbWVudDY5Njg5MzI0NA==,9599,simonw,2020-09-22T18:14:33Z,2020-09-22T18:14:45Z,OWNER,"Thinking more about this one: ``` $ sqlite-utils extract my.db \ dea_sales company_name company_address \ --table companies ``` The goal here is to pull the company name and address pair out into a separate table. Some questions: - should this first verify that every company_name has just one company_address? I like the idea of a unique constraint on the created table for this. - what should the foreign key column that gets added to the `companies` table be called?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",470345929,"table.extract(...) method and ""sqlite-utils extract"" command", https://github.com/simonw/sqlite-utils/issues/42#issuecomment-513262013,https://api.github.com/repos/simonw/sqlite-utils/issues/42,513262013,MDEyOklzc3VlQ29tbWVudDUxMzI2MjAxMw==,9599,simonw,2019-07-19T14:58:23Z,2020-09-22T18:12:11Z,OWNER,"CLI design idea: $ sqlite-utils extract my.db \ dea_sales company_name Here we just specify the original table and column - the new extracted table will automatically be called ""company_name"" and will have ""id"" and ""value"" columns, by default. To set a custom extract table: $ sqlite-utils extract my.db \ dea_sales company_name \ --table companies And for extracting multiple columns and renaming them on the created table, maybe something like this: $ sqlite-utils extract my.db \ dea_sales company_name company_address \ --table companies \ --column company_name name \ --column company_address address ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",470345929,"table.extract(...) method and ""sqlite-utils extract"" command", https://github.com/simonw/datasette/issues/973#issuecomment-696800410,https://api.github.com/repos/simonw/datasette/issues/973,696800410,MDEyOklzc3VlQ29tbWVudDY5NjgwMDQxMA==,9599,simonw,2020-09-22T15:35:28Z,2020-09-22T15:35:28Z,OWNER,"Confirmed in local dev: ``` % datasette fixtures.db --inspect-file inspect.json Traceback (most recent call last): File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/bin/datasette"", line 11, in load_entry_point('datasette', 'console_scripts', 'datasette')() File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/click/core.py"", line 829, in __call__ return self.main(*args, **kwargs) File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/click/core.py"", line 782, in main rv = self.invoke(ctx) File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/click/core.py"", line 1259, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/click/core.py"", line 1066, in invoke return ctx.invoke(self.callback, **ctx.params) File ""/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/click/core.py"", line 610, in invoke return callback(*args, **kwargs) File ""/Users/simon/Dropbox/Development/datasette/datasette/cli.py"", line 406, in serve inspect_data = json.load(open(inspect_file)) TypeError: 'bool' object is not callable ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",706486323,'bool' object is not callable error, https://github.com/simonw/datasette/issues/973#issuecomment-696798114,https://api.github.com/repos/simonw/datasette/issues/973,696798114,MDEyOklzc3VlQ29tbWVudDY5Njc5ODExNA==,9599,simonw,2020-09-22T15:31:25Z,2020-09-22T15:31:25Z,OWNER,D'oh because I have a new variable called `open`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",706486323,'bool' object is not callable error, https://github.com/simonw/datasette/issues/969#issuecomment-696788109,https://api.github.com/repos/simonw/datasette/issues/969,696788109,MDEyOklzc3VlQ29tbWVudDY5Njc4ODEwOQ==,9599,simonw,2020-09-22T15:15:14Z,2020-09-22T15:15:14Z,OWNER,"I don't think a standard ""pass these extra arguments to the publish tool"" mechanism will work because there's no guarantee that a publisher uses a CLI tool - or if it does, it might make several calls to different CLI tools. The Cloud Run one runs a couple of commands, as illustrated by this test: https://github.com/simonw/datasette/blob/a648bb82bac201c7658f6fdb499ff8ac17ebd2e8/tests/test_publish_cloudrun.py#L63-L73 Adding a `--tar` option for `datasette publish heroku` is a good fix for this though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705057955,"Add --tar option to ""datasette publish heroku""", https://github.com/simonw/datasette/issues/943#issuecomment-696778735,https://api.github.com/repos/simonw/datasette/issues/943,696778735,MDEyOklzc3VlQ29tbWVudDY5Njc3ODczNQ==,9599,simonw,2020-09-22T15:00:13Z,2020-09-22T15:00:39Z,OWNER,"Am I going to rewrite ALL of my tests to use this instead? It would clean up a lot of test code, at the cost of quite a bit of work. It would make for much neater plugin tests too, and neater testing documentation: https://docs.datasette.io/en/stable/testing_plugins.html","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681375466,await datasette.client.get(path) mechanism for executing internal requests, https://github.com/simonw/datasette/issues/943#issuecomment-696777886,https://api.github.com/repos/simonw/datasette/issues/943,696777886,MDEyOklzc3VlQ29tbWVudDY5Njc3Nzg4Ng==,9599,simonw,2020-09-22T14:58:54Z,2020-09-22T14:58:54Z,OWNER,"```python class DatasetteClient: def __init__(self, ds): self._client = httpx.AsyncClient(app=ds.app()) def _fix(self, path): if path.startswith(""/""): path = ""http://localhost{}"".format(path) return path async def get(self, path, **kwargs): return await self._client.get(self._fix(path), **kwargs) async def options(self, path, **kwargs): return await self._client.options(self._fix(path), **kwargs) async def head(self, path, **kwargs): return await self._client.head(self._fix(path), **kwargs) async def post(self, path, **kwargs): return await self._client.post(self._fix(path), **kwargs) async def put(self, path, **kwargs): return await self._client.put(self._fix(path), **kwargs) async def patch(self, path, **kwargs): return await self._client.patch(self._fix(path), **kwargs) async def delete(self, path, **kwargs): return await self._client.delete(self._fix(path), **kwargs) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681375466,await datasette.client.get(path) mechanism for executing internal requests, https://github.com/simonw/datasette/issues/943#issuecomment-696776828,https://api.github.com/repos/simonw/datasette/issues/943,696776828,MDEyOklzc3VlQ29tbWVudDY5Njc3NjgyOA==,9599,simonw,2020-09-22T14:57:13Z,2020-09-22T14:57:13Z,OWNER,"I may as well implement all of the HTTP methods supported by the `httpx` client: - get - options - head - post - put - patch - delete","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681375466,await datasette.client.get(path) mechanism for executing internal requests, https://github.com/simonw/datasette/issues/943#issuecomment-696775516,https://api.github.com/repos/simonw/datasette/issues/943,696775516,MDEyOklzc3VlQ29tbWVudDY5Njc3NTUxNg==,9599,simonw,2020-09-22T14:55:10Z,2020-09-22T14:55:10Z,OWNER,"Even smaller `DatasetteClient` implementation: ```python class DatasetteClient: def __init__(self, ds): self._client = httpx.AsyncClient(app=ds.app()) def _fix(self, path): if path.startswith(""/""): path = ""http://localhost{}"".format(path) return path async def get(self, path, **kwargs): return await self._client.get(self._fix(path), **kwargs) async def post(self, path, **kwargs): return await self._client.post(self._fix(path), **kwargs) async def options(self, path, **kwargs): return await self._client.options(self._fix(path), **kwargs) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681375466,await datasette.client.get(path) mechanism for executing internal requests, https://github.com/simonw/datasette/issues/943#issuecomment-696774711,https://api.github.com/repos/simonw/datasette/issues/943,696774711,MDEyOklzc3VlQ29tbWVudDY5Njc3NDcxMQ==,9599,simonw,2020-09-22T14:53:56Z,2020-09-22T14:53:56Z,OWNER,"How important is it to use `httpx.AsyncClient` with a context manager? https://www.python-httpx.org/async/#opening-and-closing-clients says: > Alternatively, use `await client.aclose()` if you want to close a client explicitly: > > ``` > client = httpx.AsyncClient() > ... > await client.aclose() > ``` The `.aclose()` method has a comment saying ""Close transport and proxies"" - I'm not using proxies, so the relevant implementation seems to be a call to `await self._transport.aclose()` in https://github.com/encode/httpx/blob/f932af9172d15a803ad40061a4c2c0cd891645cf/httpx/_client.py#L1741-L1751 The transport I am using is a class called `ASGITransport` in https://github.com/encode/httpx/blob/master/httpx/_transports/asgi.py The `aclose()` method on that class does nothing. So it looks like I can instantiate a client without bothering with the `async with httpx.AsyncClient` bit.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681375466,await datasette.client.get(path) mechanism for executing internal requests, https://github.com/simonw/datasette/issues/943#issuecomment-696769853,https://api.github.com/repos/simonw/datasette/issues/943,696769853,MDEyOklzc3VlQ29tbWVudDY5Njc2OTg1Mw==,9599,simonw,2020-09-22T14:46:21Z,2020-09-22T14:46:21Z,OWNER,This adds `httpx` as a dependency - I think I'm OK with that. I use it for testing in all of my plugins anyway.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681375466,await datasette.client.get(path) mechanism for executing internal requests, https://github.com/simonw/datasette/issues/943#issuecomment-696769501,https://api.github.com/repos/simonw/datasette/issues/943,696769501,MDEyOklzc3VlQ29tbWVudDY5Njc2OTUwMQ==,9599,simonw,2020-09-22T14:45:49Z,2020-09-22T14:45:49Z,OWNER,"I put together a minimal prototype of this and it feels pretty good: ```diff diff --git a/datasette/app.py b/datasette/app.py index 20aae7d..fb3bdad 100644 --- a/datasette/app.py +++ b/datasette/app.py @@ -4,6 +4,7 @@ import collections import datetime import glob import hashlib +import httpx import inspect import itertools from itsdangerous import BadSignature @@ -312,6 +313,7 @@ class Datasette: self._register_renderers() self._permission_checks = collections.deque(maxlen=200) self._root_token = secrets.token_hex(32) + self.client = DatasetteClient(self) async def invoke_startup(self): for hook in pm.hook.startup(datasette=self): @@ -1209,3 +1211,25 @@ def route_pattern_from_filepath(filepath): class NotFoundExplicit(NotFound): pass + + +class DatasetteClient: + def __init__(self, ds): + self.app = ds.app() + + def _fix(self, path): + if path.startswith(""/""): + path = ""http://localhost{}"".format(path) + return path + + async def get(self, path, **kwargs): + async with httpx.AsyncClient(app=self.app) as client: + return await client.get(self._fix(path), **kwargs) + + async def post(self, path, **kwargs): + async with httpx.AsyncClient(app=self.app) as client: + return await client.post(self._fix(path), **kwargs) + + async def options(self, path, **kwargs): + async with httpx.AsyncClient(app=self.app) as client: + return await client.options(self._fix(path), **kwargs) ``` Used like this in `ipython`: ``` In [1]: from datasette.app import Datasette In [2]: ds = Datasette([""fixtures.db""]) In [3]: (await ds.client.get(""/-/config.json"")).json() Out[3]: {'default_page_size': 100, 'max_returned_rows': 1000, 'num_sql_threads': 3, 'sql_time_limit_ms': 1000, 'default_facet_size': 30, 'facet_time_limit_ms': 200, 'facet_suggest_time_limit_ms': 50, 'hash_urls': False, 'allow_facet': True, 'allow_download': True, 'suggest_facets': True, 'default_cache_ttl': 5, 'default_cache_ttl_hashed': 31536000, 'cache_size_kb': 0, 'allow_csv_stream': True, 'max_csv_mb': 100, 'truncate_cells_html': 2048, 'force_https_urls': False, 'template_debug': False, 'base_url': '/'} In [4]: (await ds.client.get(""/fixtures/facetable.json?_shape=array"")).json() Out[4]: [{'pk': 1, 'created': '2019-01-14 08:00:00', 'planet_int': 1, 'on_earth': 1, 'state': 'CA', 'city_id': 1, 'neighborhood': 'Mission', 'tags': '[""tag1"", ""tag2""]', 'complex_array': '[{""foo"": ""bar""}]', 'distinct_some_null': 'one'}, {'pk': 2, 'created': '2019-01-14 08:00:00', 'planet_int': 1, 'on_earth': 1, 'state': 'CA', 'city_id': 1, 'neighborhood': 'Dogpatch', 'tags': '[""tag1"", ""tag3""]', 'complex_array': '[]', 'distinct_some_null': 'two'}, ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681375466,await datasette.client.get(path) mechanism for executing internal requests, https://github.com/simonw/datasette/issues/943#issuecomment-693009048,https://api.github.com/repos/simonw/datasette/issues/943,693009048,MDEyOklzc3VlQ29tbWVudDY5MzAwOTA0OA==,9599,simonw,2020-09-15T22:17:30Z,2020-09-22T14:37:00Z,OWNER,"Maybe instead of implementing `datasette.get()` and `datasette.post()` and `datasette.request()` and `datasette.stream()` I could instead have a nested object called `datasette.client` which is a preconfigured `AsyncClient` instance. ```python response = await datasette.client.get(""/"") ``` Or perhaps this should be a method in case I ever need to be able to `await` it: ```python response = await (await datasette.client()).get(""/"") ``` This is a bit cosmetically ugly though, I'd rather avoid that if possible. Maybe I could get this working by returning an object from `.client()` which provides a `await obj.get()` method: ```python response = await datasette.client().get(""/"") ``` I don't think there's any benefit to that over `await datasette.client.get()` though.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681375466,await datasette.client.get(path) mechanism for executing internal requests, https://github.com/simonw/sqlite-utils/issues/168#issuecomment-696573944,https://api.github.com/repos/simonw/sqlite-utils/issues/168,696573944,MDEyOklzc3VlQ29tbWVudDY5NjU3Mzk0NA==,9599,simonw,2020-09-22T08:11:30Z,2020-09-22T08:11:30Z,OWNER,Huh... maybe I don't need to do anything here? It looks like it's been kept up to date: https://github.com/Homebrew/homebrew-core/commits/master/Formula/sqlite-utils.rb,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",706167456,Automate (as much as possible) updates published to Homebrew, https://github.com/simonw/sqlite-utils/issues/164#issuecomment-696567988,https://api.github.com/repos/simonw/sqlite-utils/issues/164,696567988,MDEyOklzc3VlQ29tbWVudDY5NjU2Nzk4OA==,9599,simonw,2020-09-22T07:57:50Z,2020-09-22T07:57:50Z,OWNER,Documentation: https://sqlite-utils.readthedocs.io/en/latest/cli.html#transforming-tables,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",706017416,sqlite-utils transform sub-command, https://github.com/simonw/sqlite-utils/issues/42#issuecomment-696567460,https://api.github.com/repos/simonw/sqlite-utils/issues/42,696567460,MDEyOklzc3VlQ29tbWVudDY5NjU2NzQ2MA==,9599,simonw,2020-09-22T07:56:42Z,2020-09-22T07:56:42Z,OWNER,`.transform()` has landed now which should make this a lot easier to solve.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",470345929,"table.extract(...) method and ""sqlite-utils extract"" command", https://github.com/simonw/sqlite-utils/issues/26#issuecomment-696566750,https://api.github.com/repos/simonw/sqlite-utils/issues/26,696566750,MDEyOklzc3VlQ29tbWVudDY5NjU2Njc1MA==,9599,simonw,2020-09-22T07:55:00Z,2020-09-22T07:55:00Z,OWNER,"Problem: `extract` means something else now, see #47 and the upcoming work in #42.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",455486286,Mechanism for turning nested JSON into foreign keys / many-to-many, https://github.com/simonw/sqlite-utils/issues/167#issuecomment-696565981,https://api.github.com/repos/simonw/sqlite-utils/issues/167,696565981,MDEyOklzc3VlQ29tbWVudDY5NjU2NTk4MQ==,9599,simonw,2020-09-22T07:53:13Z,2020-09-22T07:53:13Z,OWNER,"Confirmed this is a bug, https://www.sqlite.org/lang_altertable.html#making_other_kinds_of_table_schema_changes explicitly says you should do the `PRAGMA foreign_keys` bits before and after the transaction, not during. Right now my code does this INSIDE the transaction: https://github.com/simonw/sqlite-utils/blob/f29f6821f2d08e91c5c6d65d885a1bbc0c743bdd/sqlite_utils/db.py#L790-L793 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",706098005,Review the foreign key pragma stuff, https://github.com/simonw/sqlite-utils/issues/164#issuecomment-696520928,https://api.github.com/repos/simonw/sqlite-utils/issues/164,696520928,MDEyOklzc3VlQ29tbWVudDY5NjUyMDkyOA==,9599,simonw,2020-09-22T05:50:17Z,2020-09-22T05:50:17Z,OWNER,"Idea for CLI options: ``` --type age integer --drop colname --rename oldname newname --not-null col --not-null-false col --pk new_id --pk-none --default col value --default-none column --drop-foreign-key col other_table other_column ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",706017416,sqlite-utils transform sub-command, https://github.com/simonw/sqlite-utils/issues/164#issuecomment-696500922,https://api.github.com/repos/simonw/sqlite-utils/issues/164,696500922,MDEyOklzc3VlQ29tbWVudDY5NjUwMDkyMg==,9599,simonw,2020-09-22T04:22:40Z,2020-09-22T04:22:40Z,OWNER,Documentation for the `.transform()` method #114 (now landed) is here: https://sqlite-utils.readthedocs.io/en/latest/python-api.html#transforming-a-table,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",706017416,sqlite-utils transform sub-command, https://github.com/simonw/sqlite-utils/issues/114#issuecomment-696500767,https://api.github.com/repos/simonw/sqlite-utils/issues/114,696500767,MDEyOklzc3VlQ29tbWVudDY5NjUwMDc2Nw==,9599,simonw,2020-09-22T04:21:45Z,2020-09-22T04:21:45Z,OWNER,Documentation: https://sqlite-utils.readthedocs.io/en/latest/python-api.html#transforming-a-table,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621989740,table.transform() method for advanced alter table, https://github.com/simonw/sqlite-utils/pull/161#issuecomment-696494070,https://api.github.com/repos/simonw/sqlite-utils/issues/161,696494070,MDEyOklzc3VlQ29tbWVudDY5NjQ5NDA3MA==,9599,simonw,2020-09-22T03:48:58Z,2020-09-22T03:48:58Z,OWNER,"One last thing. https://www.sqlite.org/lang_altertable.html#making_other_kinds_of_table_schema_change says that the first step should be: > If foreign key constraints are enabled, disable them using PRAGMA foreign_keys=OFF. And the last steps should be: > If foreign key constraints were originally enabled then run PRAGMA foreign_key_check to verify that the schema change did not break any foreign key constraints. > > Commit the transaction started in step 2. > > If foreign keys constraints were originally enabled, reenable them now. I need to implement that.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705975133,table.transform() method, https://github.com/simonw/sqlite-utils/pull/161#issuecomment-696490851,https://api.github.com/repos/simonw/sqlite-utils/issues/161,696490851,MDEyOklzc3VlQ29tbWVudDY5NjQ5MDg1MQ==,9599,simonw,2020-09-22T03:33:54Z,2020-09-22T03:33:54Z,OWNER,It would be neat if `.transform(pk=None)` converted a primary key table to a rowid table.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705975133,table.transform() method, https://github.com/simonw/sqlite-utils/pull/161#issuecomment-696488201,https://api.github.com/repos/simonw/sqlite-utils/issues/161,696488201,MDEyOklzc3VlQ29tbWVudDY5NjQ4ODIwMQ==,9599,simonw,2020-09-22T03:21:16Z,2020-09-22T03:21:16Z,OWNER,Just needs documentation now.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705975133,table.transform() method, https://github.com/simonw/sqlite-utils/pull/161#issuecomment-696485791,https://api.github.com/repos/simonw/sqlite-utils/issues/161,696485791,MDEyOklzc3VlQ29tbWVudDY5NjQ4NTc5MQ==,9599,simonw,2020-09-22T03:10:15Z,2020-09-22T03:10:15Z,OWNER,"Design decision needed on foreign keys: what does the syntax look like for removing an existing foreign key? Since I already have a good implementation of `add_foreign_key()` I'm tempted to only support dropping them. Maybe like this: ```python table.transform(drop_foreign_keys=[(""author_id"", ""author"", ""id"")]) ``` It's a bit crufty but it's such a rare use-case that I think this will be good enough.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705975133,table.transform() method, https://github.com/simonw/sqlite-utils/pull/161#issuecomment-696480925,https://api.github.com/repos/simonw/sqlite-utils/issues/161,696480925,MDEyOklzc3VlQ29tbWVudDY5NjQ4MDkyNQ==,9599,simonw,2020-09-22T02:45:47Z,2020-09-22T02:45:47Z,OWNER,"I'm not going to do `conversions=` because it would be inconsistent with how they work elsewhere. The SQL generated by this function looks like this: INSERT INTO dogs_new_tmp VALUES (a, b) SELECT a, b from dogs; So passing `conversions={""name"": ""upper(?)""})` wouldn't make sense, since we're not using arguments hence there is no-where for that `?` to go.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705975133,table.transform() method, https://github.com/simonw/sqlite-utils/issues/164#issuecomment-696473559,https://api.github.com/repos/simonw/sqlite-utils/issues/164,696473559,MDEyOklzc3VlQ29tbWVudDY5NjQ3MzU1OQ==,9599,simonw,2020-09-22T02:10:37Z,2020-09-22T02:10:37Z,OWNER,"Maybe something like this: sqlite-utils transform mydb.db mytable -c age integer --rename age dog_age ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",706017416,sqlite-utils transform sub-command, https://github.com/simonw/sqlite-utils/issues/163#issuecomment-696465788,https://api.github.com/repos/simonw/sqlite-utils/issues/163,696465788,MDEyOklzc3VlQ29tbWVudDY5NjQ2NTc4OA==,9599,simonw,2020-09-22T01:33:04Z,2020-09-22T01:33:04Z,OWNER,This would apply to `.transform()` in #114 too.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",706001517,Idea: conversions= could take Python functions, https://github.com/simonw/sqlite-utils/issues/114#issuecomment-696454485,https://api.github.com/repos/simonw/sqlite-utils/issues/114,696454485,MDEyOklzc3VlQ29tbWVudDY5NjQ1NDQ4NQ==,9599,simonw,2020-09-22T00:42:35Z,2020-09-22T00:42:35Z,OWNER,The reason I'm working on this now is that I'd like to support many more options for data cleanup in the Datasette ecosystem - so being able to do things like convert the type of existing columns becomes increasingly important.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621989740,table.transform() method for advanced alter table, https://github.com/simonw/sqlite-utils/issues/162#issuecomment-696454084,https://api.github.com/repos/simonw/sqlite-utils/issues/162,696454084,MDEyOklzc3VlQ29tbWVudDY5NjQ1NDA4NA==,9599,simonw,2020-09-22T00:40:44Z,2020-09-22T00:40:44Z,OWNER,Documentation: https://sqlite-utils.readthedocs.io/en/latest/python-api.html#registering-custom-sql-functions,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705995722,A decorator for registering custom SQL functions, https://github.com/simonw/sqlite-utils/issues/162#issuecomment-696449345,https://api.github.com/repos/simonw/sqlite-utils/issues/162,696449345,MDEyOklzc3VlQ29tbWVudDY5NjQ0OTM0NQ==,9599,simonw,2020-09-22T00:22:46Z,2020-09-22T00:22:46Z,OWNER,Inspired by the idea of adding `conversions=` to #114 - since this would make it easy to register custom Python functions that can be used to convert the values in a table.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705995722,A decorator for registering custom SQL functions, https://github.com/simonw/sqlite-utils/pull/161#issuecomment-696446658,https://api.github.com/repos/simonw/sqlite-utils/issues/161,696446658,MDEyOklzc3VlQ29tbWVudDY5NjQ0NjY1OA==,9599,simonw,2020-09-22T00:13:55Z,2020-09-22T00:14:21Z,OWNER,"Idea: allow a `conversions=` parameter, as seen on `.insert_all()` and friends, which lets you apply a SQL transformation function as part of the operation. E.g.: ```python table.transform({""age"": int}, conversions={""name"": ""upper(?)""}) ``` https://sqlite-utils.readthedocs.io/en/stable/python-api.html#converting-column-values-using-sql-functions","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705975133,table.transform() method, https://github.com/simonw/sqlite-utils/pull/161#issuecomment-696445766,https://api.github.com/repos/simonw/sqlite-utils/issues/161,696445766,MDEyOklzc3VlQ29tbWVudDY5NjQ0NTc2Ng==,9599,simonw,2020-09-22T00:10:50Z,2020-09-22T00:11:12Z,OWNER,"A less horrible interface might be the following: ```python # Ensure the 'age' column is not null: table.transform(not_null={""age""}) # The 'age' column is not null but I don't want it to be: table.transform(not_null={""age"": False}) ``` So if the argument is a set it means ""make sure these are all not null"" - if the argument is a dictionary it means ""set these to be null or not null depending on if their dictionary value is true or false"".","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705975133,table.transform() method, https://github.com/simonw/sqlite-utils/pull/161#issuecomment-696444842,https://api.github.com/repos/simonw/sqlite-utils/issues/161,696444842,MDEyOklzc3VlQ29tbWVudDY5NjQ0NDg0Mg==,9599,simonw,2020-09-22T00:07:43Z,2020-09-22T00:09:05Z,OWNER,"Syntax challenge: I could use `.transform(defaults={""age"": None})` to indicate that the `age` column should have its default removed, but how would I tell `.transform()` that the `age` column, currently `not null`, should have the `not null` removed from it? I could do this: `.transform(not_not_null={""age""})` - it's a bit gross but it's also kind of funny. I actually like it!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705975133,table.transform() method, https://github.com/simonw/sqlite-utils/pull/161#issuecomment-696444353,https://api.github.com/repos/simonw/sqlite-utils/issues/161,696444353,MDEyOklzc3VlQ29tbWVudDY5NjQ0NDM1Mw==,9599,simonw,2020-09-22T00:06:12Z,2020-09-22T00:06:12Z,OWNER,I should support `not_null=` and `default=` arguments to the `.transform()` method because it looks like you can't use `ALTER TABLE` to change those.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705975133,table.transform() method, https://github.com/simonw/sqlite-utils/pull/161#issuecomment-696443845,https://api.github.com/repos/simonw/sqlite-utils/issues/161,696443845,MDEyOklzc3VlQ29tbWVudDY5NjQ0Mzg0NQ==,9599,simonw,2020-09-22T00:04:31Z,2020-09-22T00:04:44Z,OWNER,"Good news: the `.columns` introspection does tell me those things: ``` >>> import sqlite_utils >>> db = sqlite_utils.Database(memory=True) >>> db.create_table(""foo"", {""id"": int, ""name"": str, ""age"": int}, defaults={""age"": 1}, not_null={""name"", ""age""})
>>> db[""foo""]
>>> print(db[""foo""].schema) CREATE TABLE [foo] ( [id] INTEGER, [name] TEXT NOT NULL, [age] INTEGER NOT NULL DEFAULT 1 ) >>> db[""foo""].columns [Column(cid=0, name='id', type='INTEGER', notnull=0, default_value=None, is_pk=0), Column(cid=1, name='name', type='TEXT', notnull=1, default_value=None, is_pk=0), Column(cid=2, name='age', type='INTEGER', notnull=1, default_value='1', is_pk=0)] ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705975133,table.transform() method, https://github.com/simonw/sqlite-utils/pull/161#issuecomment-696443190,https://api.github.com/repos/simonw/sqlite-utils/issues/161,696443190,MDEyOklzc3VlQ29tbWVudDY5NjQ0MzE5MA==,9599,simonw,2020-09-22T00:02:22Z,2020-09-22T00:02:22Z,OWNER,How would I detect which columns are `not_null` and what their defaults are? I don`t think my introspection logic handles that yet.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705975133,table.transform() method, https://github.com/simonw/sqlite-utils/pull/161#issuecomment-696443042,https://api.github.com/repos/simonw/sqlite-utils/issues/161,696443042,MDEyOklzc3VlQ29tbWVudDY5NjQ0MzA0Mg==,9599,simonw,2020-09-22T00:01:50Z,2020-09-22T00:01:50Z,OWNER,"When you transform a table, it should keep its primary key, foreign keys, not_null and defaults. I don't think it needs to care about `hash_id` or `extracts=` since those don't affect the structure of the table as it is being created - well, `hash_id` does but if we are transforming an existing table we will get the `hash_id` column for free.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705975133,table.transform() method, https://github.com/simonw/sqlite-utils/pull/161#issuecomment-696442621,https://api.github.com/repos/simonw/sqlite-utils/issues/161,696442621,MDEyOklzc3VlQ29tbWVudDY5NjQ0MjYyMQ==,9599,simonw,2020-09-22T00:00:23Z,2020-09-22T00:00:23Z,OWNER,I still need to figure out what to do about these various other table properties: https://github.com/simonw/sqlite-utils/blob/b34c9b40c206d7a9d7ee57a8c1f198ff1f522735/sqlite_utils/db.py#L775-L787,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705975133,table.transform() method, https://github.com/simonw/sqlite-utils/issues/114#issuecomment-696435194,https://api.github.com/repos/simonw/sqlite-utils/issues/114,696435194,MDEyOklzc3VlQ29tbWVudDY5NjQzNTE5NA==,9599,simonw,2020-09-21T23:34:14Z,2020-09-21T23:35:00Z,OWNER,"I think the fiddliest part of the implementation here is code that takes the existing `columns_dict` of the table and the incoming `columns=` and `drop=` and `rename=` parameters and produces the columns dictionary for the new table, ready to be fed to `.create_table()`. This logic probably also needs to return a structure that can be used to build the `INSERT INTO ... SELECT ... FROM` query.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621989740,table.transform() method for advanced alter table, https://github.com/simonw/sqlite-utils/issues/114#issuecomment-696434638,https://api.github.com/repos/simonw/sqlite-utils/issues/114,696434638,MDEyOklzc3VlQ29tbWVudDY5NjQzNDYzOA==,9599,simonw,2020-09-21T23:32:26Z,2020-09-21T23:32:26Z,OWNER,A test that confirms that this mechanism can turn a `rowid` into a non-rowid table would be good too.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621989740,table.transform() method for advanced alter table, https://github.com/simonw/sqlite-utils/issues/114#issuecomment-696434237,https://api.github.com/repos/simonw/sqlite-utils/issues/114,696434237,MDEyOklzc3VlQ29tbWVudDY5NjQzNDIzNw==,9599,simonw,2020-09-21T23:31:07Z,2020-09-21T23:31:57Z,OWNER,"Does it make sense to support the `pk=` argument for changing the primary key? If the user requests a primary key that doesn't make sense I think an integrity error will be raised when the SQL is being executed, which should hopefully cancel the transaction and raise an error. Need to check that this is what happens.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621989740,table.transform() method for advanced alter table, https://github.com/simonw/sqlite-utils/issues/114#issuecomment-696434097,https://api.github.com/repos/simonw/sqlite-utils/issues/114,696434097,MDEyOklzc3VlQ29tbWVudDY5NjQzNDA5Nw==,9599,simonw,2020-09-21T23:30:40Z,2020-09-21T23:30:40Z,OWNER,"Since I have a `column_order=None` argument already, maybe I can ignore the order of the columns in that first argument and use that instead?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621989740,table.transform() method for advanced alter table, https://github.com/simonw/sqlite-utils/issues/114#issuecomment-696433778,https://api.github.com/repos/simonw/sqlite-utils/issues/114,696433778,MDEyOklzc3VlQ29tbWVudDY5NjQzMzc3OA==,9599,simonw,2020-09-21T23:29:39Z,2020-09-21T23:29:39Z,OWNER,"The `columns=` argument is optional - so you can do just a rename operation like so: ``` table.transform(rename={""age"": ""dog_age""}) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621989740,table.transform() method for advanced alter table, https://github.com/simonw/sqlite-utils/issues/114#issuecomment-696433542,https://api.github.com/repos/simonw/sqlite-utils/issues/114,696433542,MDEyOklzc3VlQ29tbWVudDY5NjQzMzU0Mg==,9599,simonw,2020-09-21T23:28:58Z,2020-09-21T23:28:58Z,OWNER,"If you want to both change the type of a column AND rename it in the same operation, how would you do that? I think like this: ```python table.transform({""age"": int}, rename={""age"": ""dog_age""}) ``` So any rename logic is applied at the end, after the type transformation or re-ordering logic.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621989740,table.transform() method for advanced alter table, https://github.com/simonw/sqlite-utils/issues/114#issuecomment-696432690,https://api.github.com/repos/simonw/sqlite-utils/issues/114,696432690,MDEyOklzc3VlQ29tbWVudDY5NjQzMjY5MA==,9599,simonw,2020-09-21T23:26:32Z,2020-09-21T23:27:38Z,OWNER,"To expand on what that first argument - the `columns` argument - does. Say you have a table like this: ``` id integer name text age text ``` Any columns omitted from the `columns=` argument are left alone - they have to be explicitly dropped using `drop=` if you want to drop them. Any new columns are added (at the end of the table): ``` table.tranform({""size"": float}) ``` Any columns that have their type changed will have their type changed: ``` table.tranform({""age"": int}) ``` Should I also re-order columns if the order doesn't match? I think so. Open question as to what happens to columns that aren't mentioned at all in the dictionary though - what order should they go in?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621989740,table.transform() method for advanced alter table, https://github.com/simonw/sqlite-utils/issues/114#issuecomment-696431058,https://api.github.com/repos/simonw/sqlite-utils/issues/114,696431058,MDEyOklzc3VlQ29tbWVudDY5NjQzMTA1OA==,9599,simonw,2020-09-21T23:21:37Z,2020-09-21T23:21:37Z,OWNER,I may need to do something special for `rowid` tables to ensure that the `rowid` values in the transformed table match those from the old table.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621989740,table.transform() method for advanced alter table, https://github.com/simonw/sqlite-utils/issues/114#issuecomment-696430843,https://api.github.com/repos/simonw/sqlite-utils/issues/114,696430843,MDEyOklzc3VlQ29tbWVudDY5NjQzMDg0Mw==,9599,simonw,2020-09-21T23:21:00Z,2020-09-21T23:21:00Z,OWNER,"For FTS tables associated with the table that is being transformed, should I automatically drop the old FTS table and recreate it against the new one or will it just magically continue to work after the table is renamed?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621989740,table.transform() method for advanced alter table, https://github.com/simonw/sqlite-utils/issues/114#issuecomment-696423138,https://api.github.com/repos/simonw/sqlite-utils/issues/114,696423138,MDEyOklzc3VlQ29tbWVudDY5NjQyMzEzOA==,9599,simonw,2020-09-21T22:59:17Z,2020-09-21T23:01:06Z,OWNER,I'm going to sketch out a prototype of this new API design in that branch.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621989740,table.transform() method for advanced alter table, https://github.com/simonw/sqlite-utils/issues/114#issuecomment-696423066,https://api.github.com/repos/simonw/sqlite-utils/issues/114,696423066,MDEyOklzc3VlQ29tbWVudDY5NjQyMzA2Ng==,9599,simonw,2020-09-21T22:59:01Z,2020-09-21T22:59:01Z,OWNER,"I'm rethinking the API design now. Maybe it could look like this: To change the type of the `author_id` column from `text` to `int`: ```python books.transform({""author_id"": int}) ``` This would leave the existing columns alone, but would change the type of this column. To rename `author_id` to `author_identifier`: ```python books.transform(rename={""author_id"": ""author_identifier""}) ``` To drop a column: ```python books.transform(drop=[""author_id""]) ``` Since the parameters all operate on columns they don't need to be called `drop_column` and `rename_column`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621989740,table.transform() method for advanced alter table, https://github.com/simonw/sqlite-utils/issues/114#issuecomment-696421240,https://api.github.com/repos/simonw/sqlite-utils/issues/114,696421240,MDEyOklzc3VlQ29tbWVudDY5NjQyMTI0MA==,9599,simonw,2020-09-21T22:53:48Z,2020-09-21T22:53:48Z,OWNER,"I've decided to call this `table.transform()` - I was over-thinking whether people would remember that `.transform()` actually transforms the table, but that's what documentation is for.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",621989740,table.transform() method for advanced alter table, https://github.com/simonw/datasette/issues/972#issuecomment-696308847,https://api.github.com/repos/simonw/datasette/issues/972,696308847,MDEyOklzc3VlQ29tbWVudDY5NjMwODg0Nw==,9599,simonw,2020-09-21T19:01:25Z,2020-09-21T19:01:25Z,OWNER,I did a bunch of initial work for this in #427.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705840673,Support faceting against arbitrary SQL queries, https://github.com/simonw/datasette/issues/971#issuecomment-696307922,https://api.github.com/repos/simonw/datasette/issues/971,696307922,MDEyOklzc3VlQ29tbWVudDY5NjMwNzkyMg==,9599,simonw,2020-09-21T18:59:52Z,2020-09-21T19:00:02Z,OWNER,"Given `dbstat` isn't as widely available as I thought I'm going to let people who want to use `dbstat` run their own `select * from dbstat` queries rather than bake support directly into Datasette. The experience of exploring `dbstat` will improve if I land support for running facets against arbitrary custom SQL queries, which is half-done in that facets now execute against wrapped subqueries as-of ea66c45df96479ef66a89caa71fff1a97a862646 https://github.com/simonw/datasette/blob/ea66c45df96479ef66a89caa71fff1a97a862646/datasette/facets.py#L192-L200","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705827457,Support the dbstat table, https://github.com/simonw/datasette/issues/971#issuecomment-696304108,https://api.github.com/repos/simonw/datasette/issues/971,696304108,MDEyOklzc3VlQ29tbWVudDY5NjMwNDEwOA==,9599,simonw,2020-09-21T18:52:50Z,2020-09-21T18:52:50Z,OWNER,Looks like the `pysqlite3-binary` package doesn't support `dbstat` either.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705827457,Support the dbstat table, https://github.com/simonw/datasette/issues/971#issuecomment-696302868,https://api.github.com/repos/simonw/datasette/issues/971,696302868,MDEyOklzc3VlQ29tbWVudDY5NjMwMjg2OA==,9599,simonw,2020-09-21T18:50:40Z,2020-09-21T18:50:40Z,OWNER,Easiest way to get this may be to run `create view dbstat_view as select * from dbstat` on databases that support it.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705827457,Support the dbstat table, https://github.com/simonw/datasette/issues/971#issuecomment-696302020,https://api.github.com/repos/simonw/datasette/issues/971,696302020,MDEyOklzc3VlQ29tbWVudDY5NjMwMjAyMA==,9599,simonw,2020-09-21T18:49:09Z,2020-09-21T18:49:09Z,OWNER,... made harder to work on because I apparently don't have the `DBSTAT_VTAB` module on macOS.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705827457,Support the dbstat table, https://github.com/simonw/datasette/issues/971#issuecomment-696298614,https://api.github.com/repos/simonw/datasette/issues/971,696298614,MDEyOklzc3VlQ29tbWVudDY5NjI5ODYxNA==,9599,simonw,2020-09-21T18:43:07Z,2020-09-21T18:43:07Z,OWNER,"Or, do this: ```sql SELECT 1 FROM dbstat limit 1; ``` And see if it returns a ""table does not exist"" error.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705827457,Support the dbstat table, https://github.com/simonw/datasette/issues/971#issuecomment-696297930,https://api.github.com/repos/simonw/datasette/issues/971,696297930,MDEyOklzc3VlQ29tbWVudDY5NjI5NzkzMA==,9599,simonw,2020-09-21T18:41:47Z,2020-09-21T18:41:47Z,OWNER,"https://www.sqlite.org/dbstat.html > The DBSTAT virtual table is an eponymous virtual table, meaning that is not necessary to run CREATE VIRTUAL TABLE to create an instance of the dbstat virtual table before using it.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705827457,Support the dbstat table, https://github.com/simonw/datasette/issues/971#issuecomment-696297601,https://api.github.com/repos/simonw/datasette/issues/971,696297601,MDEyOklzc3VlQ29tbWVudDY5NjI5NzYwMQ==,9599,simonw,2020-09-21T18:41:07Z,2020-09-21T18:41:07Z,OWNER,"How to detect it? Looks like it's visible in SQLite compile time options: https://latest.datasette.io/-/versions ``` ""compile_options"": [ ""COMPILER=gcc-8.3.0"", ""ENABLE_COLUMN_METADATA"", ""ENABLE_DBSTAT_VTAB"", ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705827457,Support the dbstat table, https://github.com/simonw/datasette/issues/670#issuecomment-696163452,https://api.github.com/repos/simonw/datasette/issues/670,696163452,MDEyOklzc3VlQ29tbWVudDY5NjE2MzQ1Mg==,652285,snth,2020-09-21T14:46:10Z,2020-09-21T14:46:10Z,NONE,I'm currently using PostgREST to serve OpenAPI APIs off Postgresql databases. I would like to try out datasette once this becomes available on Postgres.,"{""total_count"": 2, ""+1"": 2, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",564833696,Prototoype for Datasette on PostgreSQL, https://github.com/simonw/datasette/issues/970#issuecomment-695896557,https://api.github.com/repos/simonw/datasette/issues/970,695896557,MDEyOklzc3VlQ29tbWVudDY5NTg5NjU1Nw==,9599,simonw,2020-09-21T04:40:12Z,2020-09-21T04:40:12Z,OWNER,The Python standard library has a module for this: https://docs.python.org/3/library/webbrowser.html,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705108492,"request an ""-o"" option on ""datasette server"" to open the default browser at the running url", https://github.com/simonw/datasette/issues/970#issuecomment-695895960,https://api.github.com/repos/simonw/datasette/issues/970,695895960,MDEyOklzc3VlQ29tbWVudDY5NTg5NTk2MA==,9599,simonw,2020-09-21T04:36:45Z,2020-09-21T04:36:45Z,OWNER,I like this. It could work with the `--root` option too and automatically sign you in as the root user.,"{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 1, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705108492,"request an ""-o"" option on ""datasette server"" to open the default browser at the running url", https://github.com/dogsheep/dogsheep-beta/issues/26#issuecomment-695879531,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/26,695879531,MDEyOklzc3VlQ29tbWVudDY5NTg3OTUzMQ==,9599,simonw,2020-09-21T02:55:28Z,2020-09-21T02:55:54Z,MEMBER,"Actually for the tie-breaker it should be something like https://latest.datasette.io/fixtures?sql=select+pk%2C+created%2C+planet_int%2C+on_earth%2C+state%2C+city_id%2C+neighborhood%2C+tags%2C+complex_array%2C+distinct_some_null+from+facetable+where+%28created+%3E+%3Ap1+or+%28created+%3D+%3Ap1+and+%28%28pk+%3E+%3Ap0%29%29%29%29+order+by+created%2C+pk+limit+11&p0=10&p1=2019-01-16+08%3A00%3A00 ```sql where ( created > :p1 or ( created = :p1 and ((pk > :p0)) ) ) ``` But with `rowid` and `timestamp` in place of `pk` and `created`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705215230,Pagination, https://github.com/dogsheep/dogsheep-beta/issues/26#issuecomment-695879237,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/26,695879237,MDEyOklzc3VlQ29tbWVudDY5NTg3OTIzNw==,9599,simonw,2020-09-21T02:53:29Z,2020-09-21T02:53:29Z,MEMBER,"If previous page ended at `2018-02-11T16:32:53+00:00`: ```sql select search_index.rowid, search_index.type, search_index.key, search_index.title, search_index.category, search_index.timestamp, search_index.search_1 from search_index where date(""timestamp"") = '2018-02-11' and timestamp < '2018-02-11T16:32:53+00:00' order by search_index.timestamp desc, rowid limit 41 ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705215230,Pagination, https://github.com/dogsheep/dogsheep-beta/issues/16#issuecomment-695877627,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/16,695877627,MDEyOklzc3VlQ29tbWVudDY5NTg3NzYyNw==,9599,simonw,2020-09-21T02:42:29Z,2020-09-21T02:42:29Z,MEMBER,"Fun twist: assuming `timestamp` is always stored as UTC, I need the interface to be timezone aware so I can see e.g. everything from 4th July 2020 in the San Francisco timezone definition of 4th July 2020.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",694493566,Timeline view, https://github.com/dogsheep/dogsheep-beta/issues/26#issuecomment-695875274,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/26,695875274,MDEyOklzc3VlQ29tbWVudDY5NTg3NTI3NA==,9599,simonw,2020-09-21T02:28:58Z,2020-09-21T02:28:58Z,MEMBER,Datasette's implementation is complex because it has to support compound primary keys: https://github.com/simonw/datasette/blob/a258339a935d8d29a95940ef1db01e98bb85ae63/datasette/utils/__init__.py#L88-L114 - but that's not something that's needed for dogsheep-beta.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705215230,Pagination, https://github.com/dogsheep/dogsheep-beta/issues/26#issuecomment-695856967,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/26,695856967,MDEyOklzc3VlQ29tbWVudDY5NTg1Njk2Nw==,9599,simonw,2020-09-21T00:26:59Z,2020-09-21T00:26:59Z,MEMBER,It's a shame Datasette doesn't currently have an easy way to implement sorted-by-rank keyset-paginated using a TableView or QueryView. I'll have to do this using the custom SQL query constructed in the plugin: https://github.com/dogsheep/dogsheep-beta/blob/bed9df2b3ef68189e2e445427721a28f4e9b4887/dogsheep_beta/__init__.py#L8-L43,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705215230,Pagination, https://github.com/dogsheep/dogsheep-beta/issues/26#issuecomment-695856398,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/26,695856398,MDEyOklzc3VlQ29tbWVudDY5NTg1NjM5OA==,9599,simonw,2020-09-21T00:22:20Z,2020-09-21T00:22:20Z,MEMBER,I'm going to try for keyset pagination sorted by relevance just as a learning exercise.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705215230,Pagination, https://github.com/dogsheep/dogsheep-beta/issues/26#issuecomment-695855723,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/26,695855723,MDEyOklzc3VlQ29tbWVudDY5NTg1NTcyMw==,9599,simonw,2020-09-21T00:16:52Z,2020-09-21T00:17:53Z,MEMBER,"It feels a bit weird to implement keyset pagination against results sorted by `rank` because the ranks could change substantially if the search index gets updated while the user is paginating. I may just ignore that though. If you want reliable pagination you can get it by sorting by date. Maybe it doesn't even make sense to offer pagination if you sort by relevance?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705215230,Pagination, https://github.com/dogsheep/dogsheep-beta/issues/26#issuecomment-695855646,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/26,695855646,MDEyOklzc3VlQ29tbWVudDY5NTg1NTY0Ng==,9599,simonw,2020-09-21T00:16:11Z,2020-09-21T00:16:11Z,MEMBER,"Should I do this with offset/limit or should I do proper keyset pagination? I think keyset because then it will work well for the full search interface with no filters or search string.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705215230,Pagination, https://github.com/dogsheep/dogsheep-beta/issues/16#issuecomment-695851036,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/16,695851036,MDEyOklzc3VlQ29tbWVudDY5NTg1MTAzNg==,9599,simonw,2020-09-20T23:34:57Z,2020-09-20T23:34:57Z,MEMBER,Really basic starting point is to add facet by date.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",694493566,Timeline view, https://github.com/simonw/sqlite-utils/issues/160#issuecomment-695839557,https://api.github.com/repos/simonw/sqlite-utils/issues/160,695839557,MDEyOklzc3VlQ29tbWVudDY5NTgzOTU1Nw==,9599,simonw,2020-09-20T21:37:03Z,2020-09-20T21:37:03Z,OWNER,"Should this support `ignore=True` as well? I'm tempted to skip that - I think `replace=True` is more useful because it implies ""ignore if the options are already the same, but replace if they are different"".","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",705190723,"table.enable_fts(..., replace=True)", https://github.com/simonw/sqlite-utils/issues/42#issuecomment-695698227,https://api.github.com/repos/simonw/sqlite-utils/issues/42,695698227,MDEyOklzc3VlQ29tbWVudDY5NTY5ODIyNw==,9599,simonw,2020-09-20T04:27:26Z,2020-09-20T04:28:26Z,OWNER,This is going to need #114 (the `transform_table()` method) in order to convert string columns into integer foreign key columns.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",470345929,"table.extract(...) method and ""sqlite-utils extract"" command", https://github.com/simonw/sqlite-utils/issues/68#issuecomment-695695776,https://api.github.com/repos/simonw/sqlite-utils/issues/68,695695776,MDEyOklzc3VlQ29tbWVudDY5NTY5NTc3Ng==,9599,simonw,2020-09-20T04:25:47Z,2020-09-20T04:25:47Z,OWNER,This is a dupe of #130 ,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",531583658,Add support for porter stemming in FTS, https://github.com/simonw/datasette/issues/943#issuecomment-695133768,https://api.github.com/repos/simonw/datasette/issues/943,695133768,MDEyOklzc3VlQ29tbWVudDY5NTEzMzc2OA==,9599,simonw,2020-09-19T00:06:56Z,2020-09-19T00:07:35Z,OWNER,"[dogsheep-beta](https://github.com/dogsheep/dogsheep-beta) could do with this too. It currently [makes a call](https://github.com/dogsheep/dogsheep-beta/blob/ab36101bdae69b11af7c6bd7edee838d052e6ecf/dogsheep_beta/__init__.py#L216-L225) to `TableView` in a similar way to `datasette-graphql` in order to calculate facets. `dogsheep-beta` would benefit with a mechanism for changing the facet timeout setting during that call (as would `datasette-graphql`, see the [DatasetteSpecialConfig mechanism](https://github.com/simonw/datasette-graphql/blob/f9dc5c518b7cdc94b93873ef20069a7ea2882a95/datasette_graphql/utils.py#L516-L519) it uses).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",681375466,await datasette.client.get(path) mechanism for executing internal requests, https://github.com/dogsheep/dogsheep-beta/issues/15#issuecomment-695124698,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/15,695124698,MDEyOklzc3VlQ29tbWVudDY5NTEyNDY5OA==,9599,simonw,2020-09-18T23:17:38Z,2020-09-18T23:17:38Z,MEMBER,This can be part of the demo instance in #6.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",694136490,Add a bunch of config examples, https://github.com/dogsheep/dogsheep-beta/issues/24#issuecomment-695113871,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/24,695113871,MDEyOklzc3VlQ29tbWVudDY5NTExMzg3MQ==,9599,simonw,2020-09-18T22:30:17Z,2020-09-18T22:30:17Z,MEMBER,"I think I know what's going on here: https://github.com/dogsheep/dogsheep-beta/blob/0f1b951c5131d16f3c8559a8e4d79ed5c559e3cb/dogsheep_beta/__init__.py#L166-L171 This is a logic bug - the `compiled` variable could be the template from the previous loop!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",703970814,"the JSON object must be str, bytes or bytearray, not 'Undefined'", https://github.com/dogsheep/dogsheep-beta/issues/25#issuecomment-695109140,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/25,695109140,MDEyOklzc3VlQ29tbWVudDY5NTEwOTE0MA==,9599,simonw,2020-09-18T22:12:20Z,2020-09-18T22:12:20Z,MEMBER,Documented here: https://github.com/dogsheep/dogsheep-beta/blob/534fc9689227eba70e69a45da0cee5820bbda9e1/README.md#datasette-plugin,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",704685890,template_debug mechanism, https://github.com/dogsheep/dogsheep-beta/issues/25#issuecomment-695108895,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/25,695108895,MDEyOklzc3VlQ29tbWVudDY5NTEwODg5NQ==,9599,simonw,2020-09-18T22:11:32Z,2020-09-18T22:11:32Z,MEMBER,"I'm going to make this a new plugin configuration setting, `template_debug`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",704685890,template_debug mechanism, https://github.com/dogsheep/dogsheep-beta/issues/24#issuecomment-694557425,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/24,694557425,MDEyOklzc3VlQ29tbWVudDY5NDU1NzQyNQ==,9599,simonw,2020-09-17T23:41:01Z,2020-09-17T23:41:01Z,MEMBER,I removed all of the `json.loads()` calls and I'm still getting that `Undefined` error.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",703970814,"the JSON object must be str, bytes or bytearray, not 'Undefined'", https://github.com/dogsheep/dogsheep-beta/issues/24#issuecomment-694554584,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/24,694554584,MDEyOklzc3VlQ29tbWVudDY5NDU1NDU4NA==,9599,simonw,2020-09-17T23:31:25Z,2020-09-17T23:31:25Z,MEMBER,"I'd prefer it if errors in these template fragments were displayed as errors inline where the fragment should have been inserted, rather than 500ing the whole page - especially since the template fragments are user-provided and could have all kinds of odd errors in them which should be as easy to debug as possible.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",703970814,"the JSON object must be str, bytes or bytearray, not 'Undefined'", https://github.com/dogsheep/dogsheep-beta/issues/24#issuecomment-694553579,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/24,694553579,MDEyOklzc3VlQ29tbWVudDY5NDU1MzU3OQ==,9599,simonw,2020-09-17T23:28:37Z,2020-09-17T23:28:37Z,MEMBER,"More investigation in pdb: ``` (dogsheep-beta) dogsheep-beta % datasette . --get '/-/beta?q=pycon&sort=oldest' --pdb > /usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/json/__init__.py(341)loads() -> raise TypeError(f'the JSON object must be str, bytes or bytearray, ' (Pdb) list 336 if s.startswith('\ufeff'): 337 raise JSONDecodeError(""Unexpected UTF-8 BOM (decode using utf-8-sig)"", 338 s, 0) 339 else: 340 if not isinstance(s, (bytes, bytearray)): 341 -> raise TypeError(f'the JSON object must be str, bytes or bytearray, ' 342 f'not {s.__class__.__name__}') 343 s = s.decode(detect_encoding(s), 'surrogatepass') 344 345 if ""encoding"" in kw: 346 import warnings (Pdb) bytes (Pdb) locals()['s'] Undefined (Pdb) type(locals()['s']) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",703970814,"the JSON object must be str, bytes or bytearray, not 'Undefined'", https://github.com/dogsheep/dogsheep-beta/issues/24#issuecomment-694552681,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/24,694552681,MDEyOklzc3VlQ29tbWVudDY5NDU1MjY4MQ==,9599,simonw,2020-09-17T23:25:54Z,2020-09-17T23:25:54Z,MEMBER,"This is the template fragment it's rendering: ```html+jinja

Tweet by @{{ display.screen_name }} ({{ display.user_name }}, {{ ""{:,}"".format(display.followers_count or 0) }} followers) on {{ display.created_at }}

{{ display.full_text }}
{% if display.media_urls and json.loads(display.media_urls) %} {% for url in json.loads(display.media_urls) %} {% endfor %} {% endif %}
```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",703970814,"the JSON object must be str, bytes or bytearray, not 'Undefined'", https://github.com/dogsheep/dogsheep-beta/issues/24#issuecomment-694552393,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/24,694552393,MDEyOklzc3VlQ29tbWVudDY5NDU1MjM5Mw==,9599,simonw,2020-09-17T23:25:01Z,2020-09-17T23:25:17Z,MEMBER,"Ran `locals()` In the debugger: `{'range': , 'dict': , 'lipsum': , 'cycler': , 'joiner': , 'namespace': , 'rank': -9.383801886431414, 'rowid': 14297, 'type': 'twitter.db/tweets', 'key': '312658917933076480', 'title': 'Tweet by @chrisstreeter', 'category': 2, 'timestamp': '2013-03-15T20:17:49+00:00', 'search_1': '@simonw are you at pycon? Would love to meet you.', 'display': {'avatar_url': 'https://pbs.twimg.com/profile_images/806275088597204993/38yLHfJi_normal.jpg', 'user_name': 'Chris Streeter', 'screen_name': 'chrisstreeter', 'followers_count': 280, 'tweet_id': 312658917933076480, 'created_at': '2013-03-15T20:17:49+00:00', 'full_text': '@simonw are you at pycon? Would love to meet you.', 'media_urls_2': '[]', 'media_urls': '[]'}, 'json': }`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",703970814,"the JSON object must be str, bytes or bytearray, not 'Undefined'", https://github.com/dogsheep/dogsheep-beta/issues/24#issuecomment-694551646,https://api.github.com/repos/dogsheep/dogsheep-beta/issues/24,694551646,MDEyOklzc3VlQ29tbWVudDY5NDU1MTY0Ng==,9599,simonw,2020-09-17T23:22:48Z,2020-09-17T23:22:48Z,MEMBER,"Looks like its happening in a Jinja fragment template for one of the results: ``` /Users/simon/Dropbox/Development/dogsheep-beta/dogsheep_beta/__init__.py(169)process_results() -> output = compiled.render({**result, **{""json"": json}}) /Users/simon/.local/share/virtualenvs/dogsheep-beta-u_po4Rpj/lib/python3.8/site-packages/jinja2/asyncsupport.py(71)render() -> return original_render(self, *args, **kwargs) /Users/simon/.local/share/virtualenvs/dogsheep-beta-u_po4Rpj/lib/python3.8/site-packages/jinja2/environment.py(1090)render() -> self.environment.handle_exception() /Users/simon/.local/share/virtualenvs/dogsheep-beta-u_po4Rpj/lib/python3.8/site-packages/jinja2/environment.py(832)handle_exception() -> reraise(*rewrite_traceback_stack(source=source)) /Users/simon/.local/share/virtualenvs/dogsheep-beta-u_po4Rpj/lib/python3.8/site-packages/jinja2/_compat.py(28)reraise() -> raise value.with_traceback(tb)