github

Custom SQL query returning 101 rows (hide)

This data as json, CSV

html_urlissue_urlidnode_idusercreated_atupdated_atauthor_associationbodyreactionsissueperformed_via_github_app
https://github.com/dogsheep/twitter-to-sqlite/issues/57#issuecomment-860063190 https://api.github.com/repos/dogsheep/twitter-to-sqlite/issues/57 860063190 MDEyOklzc3VlQ29tbWVudDg2MDA2MzE5MA== 232237 2021-06-12T14:46:44Z 2021-06-12T14:46:44Z NONE I'm having the same issue (same versions of python and twitter-to-sqlite). It's the `user-timeline` command. Other commands are working. 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 907645813  
https://github.com/simonw/datasette/issues/1286#issuecomment-860047794 https://api.github.com/repos/simonw/datasette/issues/1286 860047794 MDEyOklzc3VlQ29tbWVudDg2MDA0Nzc5NA== 4068 2021-06-12T12:36:15Z 2021-06-12T12:36:15Z NONE @mroswell That is a very nice solution. I wonder if custom classes, like `col-columnName-value` could be automatically added to cells when facets on such column are enabled, to allow custom styling without having to modify templates or add custom JavaScript code. 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 849220154  
https://github.com/simonw/sqlite-utils/issues/269#issuecomment-860031217 https://api.github.com/repos/simonw/sqlite-utils/issues/269 860031217 MDEyOklzc3VlQ29tbWVudDg2MDAzMTIxNw== 4068 2021-06-12T10:01:53Z 2021-06-12T10:01:53Z NONE `sqlite-utils transform` does not allow setting the column type to boolean: ``` Error: Invalid value for '--type': 'bool' is not one of 'INTEGER', 'TEXT', 'FLOAT', 'BLOB'. ``` 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 919250621  
https://github.com/simonw/sqlite-utils/issues/270#issuecomment-860031071 https://api.github.com/repos/simonw/sqlite-utils/issues/270 860031071 MDEyOklzc3VlQ29tbWVudDg2MDAzMTA3MQ== 4068 2021-06-12T10:00:24Z 2021-06-12T10:00:24Z NONE Sure, I am sorry if my message hasn't been clear enough. I am also a new user :) At the beginning, I just call `sqlite-utils insert "$db" "$table" "$jsonfile"` to create the database. sqlite-utils convert JSON values into `TEXT`, when it tries to determine the schema automatically. I then try to transform the table to set `JSON` as type: ``` sqlite-utils transform species.sqlite species --type criteria json Usage: sqlite-utils transform [OPTIONS] PATH TABLE Try 'sqlite-utils transform --help' for help. Error: Invalid value for '--type': 'json' is not one of 'INTEGER', 'TEXT', 'FLOAT', 'BLOB'. ``` 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 919314806  
https://github.com/simonw/sqlite-utils/issues/270#issuecomment-859986489 https://api.github.com/repos/simonw/sqlite-utils/issues/270 859986489 MDEyOklzc3VlQ29tbWVudDg1OTk4NjQ4OQ== 9599 2021-06-12T02:47:12Z 2021-06-12T02:47:12Z OWNER Can you expand on what you'd like to change here? The library and CLI tool already allow JSON data to be stored in columns: - https://sqlite-utils.datasette.io/en/stable/cli.html#nested-json-values - https://sqlite-utils.datasette.io/en/stable/python-api.html#storing-json 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 919314806  
https://github.com/simonw/sqlite-utils/issues/269#issuecomment-859940977 https://api.github.com/repos/simonw/sqlite-utils/issues/269 859940977 MDEyOklzc3VlQ29tbWVudDg1OTk0MDk3Nw== 4068 2021-06-11T22:33:08Z 2021-06-11T22:33:08Z NONE `true` and `false` json values are cast to integer, which is not optimal. 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 919250621  
https://github.com/simonw/sqlite-utils/issues/268#issuecomment-859898736 https://api.github.com/repos/simonw/sqlite-utils/issues/268 859898736 MDEyOklzc3VlQ29tbWVudDg1OTg5ODczNg== 9599 2021-06-11T20:37:44Z 2021-06-11T20:37:44Z OWNER From the prototype: ``` % sqlite-utils schema 24ways.db CREATE TABLE [articles] ( [title] TEXT , [contents] TEXT , [year] TEXT , [author] TEXT , [author_slug] TEXT , [published] TEXT , [url] TEXT , [topic] TEXT ); CREATE VIRTUAL TABLE "articles_fts" USING FTS5 ( title, author, contents, content="articles" ); CREATE TABLE 'articles_fts_data'(id INTEGER PRIMARY KEY, block BLOB); CREATE TABLE 'articles_fts_idx'(segid, term, pgno, PRIMARY KEY(segid, term)) WITHOUT ROWID; CREATE TABLE 'articles_fts_docsize'(id INTEGER PRIMARY KEY, sz BLOB); CREATE TABLE 'articles_fts_config'(k PRIMARY KEY, v) WITHOUT ROWID; % sqlite-utils schema 24ways.db | sqlite3 /tmp/boo.db Error: near line 15: table 'articles_fts_data' already exists Error: near line 16: table 'articles_fts_idx' already exists Error: near line 17: table 'articles_fts_docsize' already exists Error: near line 18: table 'articles_fts_config' already exists ``` The problem here is that the `CREATE VIRTUAL TABLE "articles_fts"...` line causes those next four tables to be created - but that means that piping the output of this command into `sqlite3` in order to re-create those tables throws errors. I don't think this matters. I see this tool as more for introspection than for recreating table structures. 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 919181559  
https://github.com/simonw/sqlite-utils/issues/268#issuecomment-859895540 https://api.github.com/repos/simonw/sqlite-utils/issues/268 859895540 MDEyOklzc3VlQ29tbWVudDg1OTg5NTU0MA== 9599 2021-06-11T20:30:34Z 2021-06-11T20:30:34Z OWNER You can currently see the `sql` on the CLI using: % sqlite-utils rows fixtures.db sqlite_master -c name -c sql name sql -------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------- simple_primary_key CREATE TABLE simple_primary_key ( id varchar(30) primary key, content text ) sqlite_autoindex_simple_primary_key_1 primary_key_multiple_columns CREATE TABLE primary_key_multiple_columns ( id varchar(30) primary key, content text, content2 text ) sqlite_autoindex_primary_key_multiple_columns_1 primary_key_multiple_columns_explicit_label CREATE TABLE primary_key_multiple_columns_explicit_label ( id varchar(30) primary key, content text, content2 text ) sqlite_autoindex_primary_key_multiple_columns_explicit_label_1 compound_primary_key CREATE TABLE compound_primary_key ( pk1 varchar(30), pk2 varchar(30), content text, PRIMARY KEY (pk1, pk2) ) sqlite_autoindex_compound_primary_key_1 compound_three_primary_keys CREATE TABLE compound_three_primary_keys ( pk1 varchar(30), pk2 varchar(30), pk3 varchar(30), content text, PRIMARY KEY (pk1, pk2, pk3) ) sqlite_autoindex_compound_three_primary_keys_1 foreign_key_references CREATE TABLE foreign_key_references ( pk varchar(30) primary key, foreign_key_with_label varchar(30), foreign_key_with_no_label varchar(30), FOREIGN KEY (foreign_key_with_label) REFERENCES simple_primary_key(id), FOREIGN KEY (foreign_key_with_no_label) REFERENCES primary_key_multiple_columns(id) ) sqlite_autoindex_foreign_key_references_1 sortable CREATE TABLE sortable ( pk1 varchar(30), pk2 varchar(30), content text, sortable integer, sortable_with_nulls real, sortable_with_nulls_2 real, text text, PRIMARY KEY (pk1, pk2) ) sqlite_autoindex_sortable_1 no_primary_key CREATE TABLE no_primary_key ( content text, a text, b text, c text ) 123_starts_with_digits CREATE TABLE [123_starts_with_digits] ( content text ) paginated_view CREATE VIEW paginated_view AS SELECT content, '- ' || content || ' -' AS content_extra FROM no_primary_key Table With Space In Name CREATE TABLE "Table With Space In Name" ( pk varchar(30) primary key, content text ) sqlite_autoindex_Table With Space In Name_1 table/with/slashes.csv CREATE TABLE "table/with/slashes.csv" ( pk varchar(30) primary key, content text ) sqlite_autoindex_table/with/slashes.csv_1 complex_foreign_keys CREATE TABLE "complex_foreign_keys" ( pk varchar(30) primary key, f1 text, f2 text, f3 text, FOREIGN KEY ("f1") REFERENCES [simple_primary_key](id), FOREIGN KEY ("f2") REFERENCES [simple_primary_key](id), FOREIGN KEY ("f3") REFERENCES [simple_primary_key](id) ) sqlite_autoindex_complex_foreign_keys_1 custom_foreign_key_label CREATE TABLE "custom_foreign_key_label" ( pk varchar(30) primary key, foreign_key_with_custom_label text, FOREIGN KEY ("foreign_key_with_custom_label") REFERENCES [primary_key_multiple_columns_explicit_label](id) ) sqlite_autoindex_custom_foreign_key_label_1 units CREATE TABLE units ( pk integer primary key, distance int, frequency int ) searchable CREATE TABLE searchable ( pk integer primary key, text1 text, text2 text, [name with . and spaces] text ) searchable_fts CREATE VIRTUAL TABLE "searchable_fts" USING FTS3 (text1, text2, [name with . and spaces], content="searchable") searchable_fts_content CREATE TABLE 'searchable_fts_content'(docid INTEGER PRIMARY KEY, 'c0text1', 'c1text2', 'c2name with . and spaces', 'c3content') searchable_fts_segments CREATE TABLE 'searchable_fts_segments'(blockid INTEGER PRIMARY KEY, block BLOB) searchable_fts_segdir CREATE TABLE 'searchable_fts_segdir'(level INTEGER,idx INTEGER,start_block INTEGER,leaves_end_block INTEGER,end_block INTEGER,root BLOB,PRIMARY KEY(level, idx)) sqlite_autoindex_searchable_fts_segdir_1 select CREATE TABLE [select] ( [group] text, [having] text, [and] text ) facet_cities CREATE TABLE facet_cities ( id integer primary key, name text ) simple_view CREATE VIEW simple_view AS SELECT content, upper(content) AS upper_content FROM simple_primary_key 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 919181559  
https://github.com/simonw/sqlite-utils/issues/268#issuecomment-859894105 https://api.github.com/repos/simonw/sqlite-utils/issues/268 859894105 MDEyOklzc3VlQ29tbWVudDg1OTg5NDEwNQ== 9599 2021-06-11T20:28:52Z 2021-06-11T20:28:52Z OWNER Out of interest, here are the rows from that table where `sql` is `null`: https://latest.datasette.io/fixtures?sql=select%0D%0A++*%0D%0Afrom%0D%0A++sqlite_master%0D%0Awhere%0D%0A++sql+is+null ```csv type,name,tbl_name,rootpage,sql index,sqlite_autoindex_simple_primary_key_1,simple_primary_key,3, index,sqlite_autoindex_primary_key_multiple_columns_1,primary_key_multiple_columns,5, index,sqlite_autoindex_primary_key_multiple_columns_explicit_label_1,primary_key_multiple_columns_explicit_label,7, index,sqlite_autoindex_compound_primary_key_1,compound_primary_key,9, index,sqlite_autoindex_compound_three_primary_keys_1,compound_three_primary_keys,11, index,sqlite_autoindex_foreign_key_references_1,foreign_key_references,14, index,sqlite_autoindex_sortable_1,sortable,16, index,sqlite_autoindex_Table With Space In Name_1,Table With Space In Name,20, index,sqlite_autoindex_table/with/slashes.csv_1,table/with/slashes.csv,22, index,sqlite_autoindex_complex_foreign_keys_1,complex_foreign_keys,24, index,sqlite_autoindex_custom_foreign_key_label_1,custom_foreign_key_label,26, index,sqlite_autoindex_tags_1,tags,31, index,sqlite_autoindex_searchable_tags_1,searchable_tags,34, index,sqlite_autoindex_searchable_fts_segdir_1,searchable_fts_segdir,37, ``` 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 919181559  
https://github.com/simonw/sqlite-utils/issues/268#issuecomment-859888469 https://api.github.com/repos/simonw/sqlite-utils/issues/268 859888469 MDEyOklzc3VlQ29tbWVudDg1OTg4ODQ2OQ== 9599 2021-06-11T20:26:20Z 2021-06-11T20:26:20Z OWNER `sqlite-utils schema data.db` could output the same thing to the console. 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 919181559  
https://github.com/simonw/datasette/pull/1374#issuecomment-859572791 https://api.github.com/repos/simonw/datasette/issues/1374 859572791 MDEyOklzc3VlQ29tbWVudDg1OTU3Mjc5MQ== 22429695 2021-06-11T13:12:58Z 2021-06-11T13:12:58Z NONE # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1374?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report > Merging [#1374](https://codecov.io/gh/simonw/datasette/pull/1374?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (0ef0dd5) into [main](https://codecov.io/gh/simonw/datasette/commit/cd7678fde65319d7b6955ce9f4678ba4b9e64b66?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (cd7678f) will **not change** coverage. > The diff coverage is `n/a`. [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1374/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1374?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) ```diff @@ Coverage Diff @@ ## main #1374 +/- ## ======================================= Coverage 91.68% 91.68% ======================================= Files 34 34 Lines 4340 4340 ======================================= Hits 3979 3979 Misses 361 361 ``` ------ [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1374?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1374?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [cd7678f...0ef0dd5](https://codecov.io/gh/simonw/datasette/pull/1374?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 918730335  
https://github.com/simonw/datasette/issues/858#issuecomment-858831895 https://api.github.com/repos/simonw/datasette/issues/858 858831895 MDEyOklzc3VlQ29tbWVudDg1ODgzMTg5NQ== 56045588 2021-06-10T17:44:09Z 2021-06-10T17:44:09Z NONE any fixes for that recursive issue with temp file? I get it using both heroku and cloudrun, although it seems to still publish and deploy fine 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 642388564  
https://github.com/simonw/datasette/issues/858#issuecomment-858813675 https://api.github.com/repos/simonw/datasette/issues/858 858813675 MDEyOklzc3VlQ29tbWVudDg1ODgxMzY3NQ== 56045588 2021-06-10T17:27:46Z 2021-06-10T17:27:46Z NONE shell=True is added to line 56 (I guess it used to be 54) of heroku.py as detailed in the original issue. (for posterity) 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 642388564  
https://github.com/simonw/datasette/issues/1371#issuecomment-858099514 https://api.github.com/repos/simonw/datasette/issues/1371 858099514 MDEyOklzc3VlQ29tbWVudDg1ODA5OTUxNA== 9599 2021-06-09T21:03:49Z 2021-06-09T21:03:49Z OWNER I'll release these as an alpha straight away - it makes sense to have plugin hook changes available for people to test as alpha dependencies ASAP. 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 915455228  
https://github.com/simonw/datasette/pull/1373#issuecomment-857684605 https://api.github.com/repos/simonw/datasette/issues/1373 857684605 MDEyOklzc3VlQ29tbWVudDg1NzY4NDYwNQ== 22429695 2021-06-09T13:15:31Z 2021-06-09T13:15:31Z NONE # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1373?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report > Merging [#1373](https://codecov.io/gh/simonw/datasette/pull/1373?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (d117ba7) into [main](https://codecov.io/gh/simonw/datasette/commit/a3faf378834cc9793adeb22dee19ef57c417457e?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (a3faf37) will **not change** coverage. > The diff coverage is `n/a`. [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1373/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1373?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) ```diff @@ Coverage Diff @@ ## main #1373 +/- ## ======================================= Coverage 91.68% 91.68% ======================================= Files 34 34 Lines 4340 4340 ======================================= Hits 3979 3979 Misses 361 361 ``` ------ [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1373?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1373?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [a3faf37...d117ba7](https://codecov.io/gh/simonw/datasette/pull/1373?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 916183914  
https://github.com/simonw/datasette/pull/1370#issuecomment-857298526 https://api.github.com/repos/simonw/datasette/issues/1370 857298526 MDEyOklzc3VlQ29tbWVudDg1NzI5ODUyNg== 25778 2021-06-09T01:18:59Z 2021-06-09T01:18:59Z NONE I'm happy to grab some or all of these in this PR, if you want. 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 914130834  
https://github.com/simonw/datasette/pull/1370#issuecomment-857139881 https://api.github.com/repos/simonw/datasette/issues/1370 857139881 MDEyOklzc3VlQ29tbWVudDg1NzEzOTg4MQ== 9599 2021-06-08T20:58:41Z 2021-06-08T20:58:41Z OWNER We can remove a bunch of unnecessary `str(path)` calls too - this search finds a bunch of possible candidates: https://ripgrep.datasette.io/-/ripgrep?pattern=str%5C%28.*%28db%7Cpath%29&glob=datasette%2F**%2F*.py 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 914130834  
https://github.com/simonw/datasette/pull/1368#issuecomment-856182547 https://api.github.com/repos/simonw/datasette/issues/1368 856182547 MDEyOklzc3VlQ29tbWVudDg1NjE4MjU0Nw== 2670795 2021-06-07T18:59:47Z 2021-06-07T23:04:25Z NONE Note that if we went with a "update_metadata" hook, the hook signature would look something like this (it would return nothing): ``` update_metadata( datasette=self, metadata=metadata, key=key, database=database, table=table, fallback=fallback ) ``` The Datasette function `_metadata_recursive_update(self, orig, updated)` would disappear into the plugins. Doing this, though, we'd lose the easy ability to make the local metadata.yaml immutable (since we'd no longer have the recursive update). 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 913865304  
https://github.com/simonw/sqlite-utils/issues/266#issuecomment-856231119 https://api.github.com/repos/simonw/sqlite-utils/issues/266 856231119 MDEyOklzc3VlQ29tbWVudDg1NjIzMTExOQ== 9599 2021-06-07T20:26:05Z 2021-06-07T20:26:05Z OWNER https://github.com/python/cpython/blob/2ab27c4af4ddf7528e1375e77c787c7fbb09b5e6/Lib/typing.py#L2173-L2195 In Python 3.6 or higher can do this: ```python class Employee(NamedTuple): name: str id: int ``` 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 913135723  
https://github.com/simonw/datasette/issues/1365#issuecomment-856212136 https://api.github.com/repos/simonw/datasette/issues/1365 856212136 MDEyOklzc3VlQ29tbWVudDg1NjIxMjEzNg== 9599 2021-06-07T19:54:04Z 2021-06-07T19:54:04Z OWNER I've hit this one too. I agree, fixing this in Datasette itself is better than fixing it in the tests across multiple other projects. 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 913017577  
https://github.com/simonw/datasette/issues/1369#issuecomment-856208637 https://api.github.com/repos/simonw/datasette/issues/1369 856208637 MDEyOklzc3VlQ29tbWVudDg1NjIwODYzNw== 9599 2021-06-07T19:47:23Z 2021-06-07T19:47:23Z OWNER No point in showing the IDs twice if the blue label doesn't differ from the gray ID 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 913900374  
https://github.com/simonw/datasette/issues/1367#issuecomment-856160770 https://api.github.com/repos/simonw/datasette/issues/1367 856160770 MDEyOklzc3VlQ29tbWVudDg1NjE2MDc3MA== 9599 2021-06-07T18:22:33Z 2021-06-07T18:22:33Z OWNER Here's why: https://github.com/simonw/datasette/blob/03ec71193b9545536898a4bc7493274fec48bdd7/datasette/static/app.css#L455-L458 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 913823889  
https://github.com/simonw/datasette/issues/1366#issuecomment-856147969 https://api.github.com/repos/simonw/datasette/issues/1366 856147969 MDEyOklzc3VlQ29tbWVudDg1NjE0Nzk2OQ== 9599 2021-06-07T18:03:03Z 2021-06-07T18:03:03Z OWNER Here's an example of a test that uses it. It's necessary because sometimes fixtures that create temporary directories break in unexpected ways: https://github.com/simonw/datasette/blob/0a7621f96f8ad14da17e7172e8a7bce24ef78966/tests/test_plugins.py#L658-L666 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 913809802  
https://github.com/simonw/datasette/issues/1366#issuecomment-856147450 https://api.github.com/repos/simonw/datasette/issues/1366 856147450 MDEyOklzc3VlQ29tbWVudDg1NjE0NzQ1MA== 9599 2021-06-07T18:02:13Z 2021-06-07T18:02:13Z OWNER The hack in question is this fixture, which I've been using in an ad-hoc manner to work around errors while running the tests: https://github.com/simonw/datasette/blob/030deb4b25cda842ff7129ab7c18550c44dd8379/tests/conftest.py#L62-L75 I don't understand the underlying issue well enough to know how to get rid of it. 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 913809802  
https://github.com/simonw/sqlite-utils/issues/266#issuecomment-855611939 https://api.github.com/repos/simonw/sqlite-utils/issues/266 855611939 MDEyOklzc3VlQ29tbWVudDg1NTYxMTkzOQ== 9599 2021-06-07T06:07:41Z 2021-06-07T06:07:41Z OWNER Looks like this is the way to do this: ```python Point = typing.NamedTuple( "Point", [('x', int), ('y', int)] ) ``` 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 913135723  
https://github.com/simonw/datasette/issues/1362#issuecomment-855430317 https://api.github.com/repos/simonw/datasette/issues/1362 855430317 MDEyOklzc3VlQ29tbWVudDg1NTQzMDMxNw== 9599 2021-06-06T17:07:48Z 2021-06-06T17:07:48Z OWNER I guess I can offer a `disable_csp` setting so that people with complex custom templates aren't completely blocked from using them with Datasette, but maybe it would be better not to offer that? Or to offer it as a `datasette-insecure-csp` plugin instead? I like the idea of very actively encouraging CSP across all Datasette projects, but I'm nervous about making the software unusable for certain edge cases. Maybe require CSP and wait for someone to complain? 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 912864936  
https://github.com/simonw/datasette/issues/1362#issuecomment-855429111 https://api.github.com/repos/simonw/datasette/issues/1362 855429111 MDEyOklzc3VlQ29tbWVudDg1NTQyOTExMQ== 9599 2021-06-06T16:59:05Z 2021-06-06T17:00:15Z OWNER Twitter conversation: https://twitter.com/simonw/status/1401565566045806594 @dracos provided some really useful code examples there: > We generate it here: https://github.com/mysociety/fixmystreet/blob/e9fec4e567e7148ed128816e5770c2963be51af6/perllib/FixMyStreet/Cobrand/Default.pm#L89-L90 And use it e.g. https://github.com/mysociety/fixmystreet/blob/ba6788cd25d8f471a4e3308403607627b4d2f4f6/templates/web/base/common_header_tags.html or https://github.com/mysociety/fixmystreet/blob/cb4f2b96364d151988b5c664888468b25cc62240/templates/web/fixmystreet.com/header/css.html 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 912864936  
https://github.com/simonw/datasette/issues/1362#issuecomment-855428601 https://api.github.com/repos/simonw/datasette/issues/1362 855428601 MDEyOklzc3VlQ29tbWVudDg1NTQyODYwMQ== 9599 2021-06-06T16:55:33Z 2021-06-06T16:55:33Z OWNER > No, because Vary header is about _request_ headers that cause the response to vary, not response headers. Hah, of course! Thanks for the correction. So the nonce mechanism would actually be pretty great here, especially for the `extra_body_script()` hook. 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 912864936  
https://github.com/simonw/datasette/issues/1362#issuecomment-855428296 https://api.github.com/repos/simonw/datasette/issues/1362 855428296 MDEyOklzc3VlQ29tbWVudDg1NTQyODI5Ng== 154364 2021-06-06T16:53:20Z 2021-06-06T16:53:20Z NONE > Presumably this would also require adding Content-Security-Policy to the Vary header though, which will have a nasty effect on Cloudflare and Fastly and such like. No, because Vary header is about *request* headers that cause the response to vary, not response headers. 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 912864936  
https://github.com/simonw/datasette/issues/1362#issuecomment-855427396 https://api.github.com/repos/simonw/datasette/issues/1362 855427396 MDEyOklzc3VlQ29tbWVudDg1NTQyNzM5Ng== 9599 2021-06-06T16:46:17Z 2021-06-06T16:46:17Z OWNER Mind you, since that plugin hook looks like this: ```python @hookimpl def extra_body_script(): return { "module": True, "script": "console.log('Your JavaScript goes here...')" } ``` Having it calculate a sha256 hash wouldn't be difficult. 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 912864936  
https://github.com/simonw/datasette/issues/1362#issuecomment-855426750 https://api.github.com/repos/simonw/datasette/issues/1362 855426750 MDEyOklzc3VlQ29tbWVudDg1NTQyNjc1MA== 9599 2021-06-06T16:41:30Z 2021-06-06T16:44:49Z OWNER This is from the current `base.html` template: https://github.com/simonw/datasette/blob/030deb4b25cda842ff7129ab7c18550c44dd8379/datasette/templates/base.html#L62-L66 Which includes this: https://github.com/simonw/datasette/blob/030deb4b25cda842ff7129ab7c18550c44dd8379/datasette/templates/_close_open_menus.html#L1-L16 The `body_scripts` bit is for this `extra_body_script` plugin hook, which is the thing that will be the most affected by implementing CSP: https://docs.datasette.io/en/stable/plugin_hooks.html#extra-body-script-template-database-table-columns-view-name-request-datasette 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 912864936  
https://github.com/simonw/datasette/issues/1362#issuecomment-855426516 https://api.github.com/repos/simonw/datasette/issues/1362 855426516 MDEyOklzc3VlQ29tbWVudDg1NTQyNjUxNg== 9599 2021-06-06T16:39:34Z 2021-06-06T16:39:34Z OWNER The reason Datasette uses small inline scripts right now is to avoid the overhead of an extra HTTP request for a JavaScript file - but these are both inherently cachable and perform much better under HTTP/2 so that's likely a false optimization. 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 912864936  
https://github.com/simonw/datasette/issues/1362#issuecomment-855426314 https://api.github.com/repos/simonw/datasette/issues/1362 855426314 MDEyOklzc3VlQ29tbWVudDg1NTQyNjMxNA== 9599 2021-06-06T16:38:04Z 2021-06-06T16:38:04Z OWNER The other option for inline scripts is the CSP nonce: Content-Security-Policy: script-src 'nonce-2726c7f26c' Then: <script nonce="2726c7f26c"> var inline = 1; </script> Since an attacker can't guess what the nonce will be it prevents them from injecting their own script block - this seems easier to make available to plugins than a full hashing mechanism, just make `{{ csp_nonce() }}` available to the template. That template function can then be smart enough to set a flag which Datasette uses to decide if the `script-src 'nonce-2726c7f26c'` policy should be sent or not. Presumably this would also require adding `Content-Security-Policy` to the `Vary` header though, which will have a nasty effect on Cloudflare and Fastly and such like. 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 912864936  
https://github.com/simonw/datasette/issues/1362#issuecomment-855418899 https://api.github.com/repos/simonw/datasette/issues/1362 855418899 MDEyOklzc3VlQ29tbWVudDg1NTQxODg5OQ== 9599 2021-06-06T15:42:55Z 2021-06-06T15:42:55Z OWNER Another consideration: testing that this works correctly could require adoption of a real browser test environment (probably Cypress or maybe Playwright) to execute tests that will fail if CSP is violated. 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 912864936  
https://github.com/simonw/datasette/issues/1362#issuecomment-855418698 https://api.github.com/repos/simonw/datasette/issues/1362 855418698 MDEyOklzc3VlQ29tbWVudDg1NTQxODY5OA== 9599 2021-06-06T15:41:24Z 2021-06-06T15:41:24Z OWNER I think the best way to answer these questions is with some prototyping - of both Datasette and some of the existing JavaScript plugins. I can start with a `datasette-experimental-csp` plugin that sets the header (and could even run an optional report URI mechanism). 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 912864936  
https://github.com/simonw/datasette/issues/1362#issuecomment-855418401 https://api.github.com/repos/simonw/datasette/issues/1362 855418401 MDEyOklzc3VlQ29tbWVudDg1NTQxODQwMQ== 9599 2021-06-06T15:39:38Z 2021-06-06T15:39:38Z OWNER The security benefit of forcing all JavaScript plugins to be written as CSP-friendly external scripts is very compelling though. Other plugin-heavy ecosystems such as WordPress have suffered greatly from insecurely written plugins - could this be a huge security win for the Datasette ecosystem generally? 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 912864936  
https://github.com/simonw/datasette/issues/1362#issuecomment-855418065 https://api.github.com/repos/simonw/datasette/issues/1362 855418065 MDEyOklzc3VlQ29tbWVudDg1NTQxODA2NQ== 9599 2021-06-06T15:37:11Z 2021-06-06T15:37:11Z OWNER The easiest way to apply CSP is to remove all inline `<script>` blocks (Datasette has a few) and instead serve JavaScript as separate linked files. It's possible to keep inline script blocks by calculating a hash of their content and adding a `Content-Security-Policy: script-src 'sha256-B2yPHKaXnvFWtRChIbabYmUBFZdVfKKXHbWtWidDVF8='` to the policy. https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Security-Policy/script-src This could be achieved with some Django template tricks, but it feels very risky - and done carelessly could end up calculating a hash of a reflected XSS attack! The biggest challenge I see around here involves plugins and custom templates. Adopting CSP would require plugins to avoid using any inline scripts, instead keeping their entire implementations in `.js` files. That's maybe not a bad thing, but it represents a big commitment. It would need to be adopted before Datasette 1.0. 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 912864936  
https://github.com/simonw/datasette/issues/283#issuecomment-855369819 https://api.github.com/repos/simonw/datasette/issues/283 855369819 MDEyOklzc3VlQ29tbWVudDg1NTM2OTgxOQ== 9599 2021-06-06T09:40:18Z 2021-06-06T09:40:18Z OWNER > One note on using this pragma I got an error on starting datasette `no such table: pragma_database_list`. > > I diagnosed this to an older version of sqlite3 (3.14.2) and upgrading to a newer version (3.34.2) fixed the issue. That issue is fixed in #1276. 
{
    "total_count": 1,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 1,
    "rocket": 0,
    "eyes": 0
}
 325958506  
https://github.com/simonw/datasette/issues/1361#issuecomment-855308811 https://api.github.com/repos/simonw/datasette/issues/1361 855308811 MDEyOklzc3VlQ29tbWVudDg1NTMwODgxMQ== 9599 2021-06-05T23:16:21Z 2021-06-05T23:16:21Z OWNER That seems to have fixed it. I'd love to get rid of this `restore_working_directory` hack entirely. 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 912485040  
https://github.com/simonw/datasette/issues/1361#issuecomment-855307292 https://api.github.com/repos/simonw/datasette/issues/1361 855307292 MDEyOklzc3VlQ29tbWVudDg1NTMwNzI5Mg== 9599 2021-06-05T22:59:35Z 2021-06-05T22:59:35Z OWNER That broke things. Here's how `pytest-cov` fixed a similar issue: https://github.com/pytest-dev/pytest-cov/commit/7ccb1783bf8290447df58deeb41eae74296a6d9b See also https://github.com/nedbat/coveragepy/issues/881 and https://github.com/pytest-dev/pytest-cov/issues/306 and https://github.com/nedbat/coveragepy/issues/824 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 912485040  
https://github.com/simonw/datasette/issues/1361#issuecomment-855306497 https://api.github.com/repos/simonw/datasette/issues/1361 855306497 MDEyOklzc3VlQ29tbWVudDg1NTMwNjQ5Nw== 9599 2021-06-05T22:51:01Z 2021-06-05T22:51:01Z OWNER I'm going to try removing that `restore_working_directory` fixture entirely. 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 912485040  
https://github.com/simonw/datasette/issues/1361#issuecomment-855306347 https://api.github.com/repos/simonw/datasette/issues/1361 855306347 MDEyOklzc3VlQ29tbWVudDg1NTMwNjM0Nw== 9599 2021-06-05T22:49:30Z 2021-06-05T22:49:30Z OWNER Stack Overflow: https://stackoverflow.com/a/49367679/6083 > The answer was that `os.chdir()` had been set to the deleted directory by accident. The directory was missing, but the error happened (it seems) at the attempt to get it with `os.getcwd()`. 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 912485040  
https://github.com/simonw/datasette/issues/1360#issuecomment-855303776 https://api.github.com/repos/simonw/datasette/issues/1360 855303776 MDEyOklzc3VlQ29tbWVudDg1NTMwMzc3Ng== 9599 2021-06-05T22:23:23Z 2021-06-05T22:23:23Z OWNER Worth noting that I found this issue myself, and to my knowledge it has not been uncovered by anyone else prior to the patch being released. 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 912464443  
https://github.com/simonw/datasette/issues/1360#issuecomment-855303649 https://api.github.com/repos/simonw/datasette/issues/1360 855303649 MDEyOklzc3VlQ29tbWVudDg1NTMwMzY0OQ== 9599 2021-06-05T22:22:06Z 2021-06-05T22:22:06Z OWNER I've released fixes in both 0.56.1 and 0.57. 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 912464443  
https://github.com/simonw/datasette/issues/1358#issuecomment-855302339 https://api.github.com/repos/simonw/datasette/issues/1358 855302339 MDEyOklzc3VlQ29tbWVudDg1NTMwMjMzOQ== 9599 2021-06-05T22:08:16Z 2021-06-05T22:08:16Z OWNER Release notes are in, pushing the release now. 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 912418094  
https://github.com/simonw/datasette/issues/1358#issuecomment-855302320 https://api.github.com/repos/simonw/datasette/issues/1358 855302320 MDEyOklzc3VlQ29tbWVudDg1NTMwMjMyMA== 9599 2021-06-05T22:08:06Z 2021-06-05T22:08:06Z OWNER See #1360 for the security fix. 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 912418094  
https://github.com/simonw/datasette/issues/1358#issuecomment-855288228 https://api.github.com/repos/simonw/datasette/issues/1358 855288228 MDEyOklzc3VlQ29tbWVudDg1NTI4ODIyOA== 9599 2021-06-05T19:57:18Z 2021-06-05T19:57:18Z OWNER There's also a security fix I need to make, so I'm not going to block this on wrapping up the work on the new Docker image testing from #1344 before putting out this release. 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 912418094  
https://github.com/simonw/datasette/pull/1291#issuecomment-855287200 https://api.github.com/repos/simonw/datasette/issues/1291 855287200 MDEyOklzc3VlQ29tbWVudDg1NTI4NzIwMA== 9599 2021-06-05T19:48:36Z 2021-06-05T19:48:36Z OWNER This is great, thank you. 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 849582643  
https://github.com/simonw/datasette/pull/1291#issuecomment-812815358 https://api.github.com/repos/simonw/datasette/issues/1291 812815358 MDEyOklzc3VlQ29tbWVudDgxMjgxNTM1OA== 22429695 2021-04-03T05:32:50Z 2021-06-05T19:48:30Z NONE # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1291?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report > Merging [#1291](https://codecov.io/gh/simonw/datasette/pull/1291?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (8789157) into [main](https://codecov.io/gh/simonw/datasette/commit/0a7621f96f8ad14da17e7172e8a7bce24ef78966?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (0a7621f) will **not change** coverage. > The diff coverage is `n/a`. > :exclamation: Current head 8789157 differs from pull request most recent head 9bf089f. Consider uploading reports for the commit 9bf089f to get more accurate results [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1291/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1291?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) ```diff @@ Coverage Diff @@ ## main #1291 +/- ## ======================================= Coverage 91.51% 91.51% ======================================= Files 34 34 Lines 4255 4255 ======================================= Hits 3894 3894 Misses 361 361 ``` ------ [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1291?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1291?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [0a7621f...9bf089f](https://codecov.io/gh/simonw/datasette/pull/1291?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 849582643  
https://github.com/simonw/datasette/issues/1356#issuecomment-855282466 https://api.github.com/repos/simonw/datasette/issues/1356 855282466 MDEyOklzc3VlQ29tbWVudDg1NTI4MjQ2Ng== 9599 2021-06-05T19:05:06Z 2021-06-05T19:05:06Z OWNER Yeah that's a good point. I avoided making them sub-commands because `datasette serve` already supports the multitude of other arguments they also need... but actually that was just me being lazy - I can easily share arguments between multiple functions like I do in `sqlite-utils` itself: https://github.com/simonw/sqlite-utils/blob/d1a372b3006e6cf7d2017b3ddc484bf5c033e45d/sqlite_utils/cli.py#L46 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 910092577  
https://github.com/simonw/datasette/pull/1357#issuecomment-855281774 https://api.github.com/repos/simonw/datasette/issues/1357 855281774 MDEyOklzc3VlQ29tbWVudDg1NTI4MTc3NA== 22429695 2021-06-05T18:59:07Z 2021-06-05T18:59:07Z NONE # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1357?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report > Merging [#1357](https://codecov.io/gh/simonw/datasette/pull/1357?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (1b27643) into [main](https://codecov.io/gh/simonw/datasette/commit/6e9b07be92905011211d8df7a872fb7c1f2737b2?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (6e9b07b) will **increase** coverage by `0.00%`. > The diff coverage is `100.00%`. [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1357/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1357?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) ```diff @@ Coverage Diff @@ ## main #1357 +/- ## ======================================= Coverage 91.71% 91.72% ======================================= Files 34 34 Lines 4333 4336 +3 ======================================= + Hits 3974 3977 +3 Misses 359 359 ``` | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1357?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | | |---|---|---| | [datasette/utils/testing.py](https://codecov.io/gh/simonw/datasette/pull/1357/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3V0aWxzL3Rlc3RpbmcucHk=) | `95.38% <ø> (ø)` | | | [datasette/app.py](https://codecov.io/gh/simonw/datasette/pull/1357/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `95.71% <100.00%> (+0.01%)` | :arrow_up: | ------ [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1357?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1357?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [6e9b07b...1b27643](https://codecov.io/gh/simonw/datasette/pull/1357?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 912394511  
https://github.com/simonw/datasette/issues/1238#issuecomment-855278998 https://api.github.com/repos/simonw/datasette/issues/1238 855278998 MDEyOklzc3VlQ29tbWVudDg1NTI3ODk5OA== 9599 2021-06-05T18:37:16Z 2021-06-05T18:37:16Z OWNER Alternative idea: populate `request.scope` with a new `route_path` which is the base-url-stripped version, which we then use for other routing operations. 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 813899472  
https://github.com/simonw/datasette/issues/1238#issuecomment-855278540 https://api.github.com/repos/simonw/datasette/issues/1238 855278540 MDEyOklzc3VlQ29tbWVudDg1NTI3ODU0MA== 9599 2021-06-05T18:33:25Z 2021-06-05T18:33:25Z OWNER Got the test to pass by ensuring the tests don't accidentally double-rewrite the path. Ran into a new problem: ``` @pytest.mark.asyncio @pytest.mark.parametrize( "prefix,expected_path", [(None, "/asgi-scope"), ("/prefix/", "/prefix/asgi-scope")] ) async def test_client_path(datasette, prefix, expected_path): original_base_url = datasette._settings["base_url"] try: if prefix is not None: datasette._settings["base_url"] = prefix response = await datasette.client.get("/asgi-scope") path = response.json()["path"] > assert path == expected_path E AssertionError: assert '/asgi-scope' == '/prefix/asgi-scope' E - /prefix/asgi-scope E ? ------- E + /asgi-scope ``` That test confirms that messing around with the `base_url` doesn't modify the ASGI scope... but the fix I'm using for this issue DOES modify the ASGI scope. The question raised here is: should the ASGI scope stay unmodified when `base_url` is used? I think it should. It doesn't make sense to obscure the "real" path just to get custom pages to work properly. 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 813899472  
https://github.com/simonw/datasette/issues/1238#issuecomment-855272693 https://api.github.com/repos/simonw/datasette/issues/1238 855272693 MDEyOklzc3VlQ29tbWVudDg1NTI3MjY5Mw== 9599 2021-06-05T17:45:42Z 2021-06-05T17:45:42Z OWNER Applying this fix worked when I manually tested it: ```diff base_url = self.ds.setting("base_url") if base_url != "/" and path.startswith(base_url): path = "/" + path[len(base_url) :] + scope = dict(scope, path=path, raw_path=path.encode("utf-8")) request = Request(scope, receive) ``` But... the test I wrote still failed. My hunch is that this is because deep within the test framework requests go through `ds.client` which may be applying its own changes relevant to `base_url`: https://github.com/simonw/datasette/blob/6e9b07be92905011211d8df7a872fb7c1f2737b2/datasette/utils/testing.py#L139 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 813899472  
https://github.com/simonw/datasette/issues/1238#issuecomment-855270917 https://api.github.com/repos/simonw/datasette/issues/1238 855270917 MDEyOklzc3VlQ29tbWVudDg1NTI3MDkxNw== 9599 2021-06-05T17:32:29Z 2021-06-05T17:32:29Z OWNER This looks like the cause: https://github.com/simonw/datasette/blob/6e9b07be92905011211d8df7a872fb7c1f2737b2/datasette/app.py#L1087-L1092 Note how `path` is modified... but then we create a new `Request()` that uses the old scope, which has unmodified `scope["path"]` - and then the code later on looks at `request.scope["path"]` when deciding if the request matches: https://github.com/simonw/datasette/blob/afed51b1e36cf275c39e71c7cb262d6c5bdbaa31/datasette/app.py#L1154-L1155 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 813899472  
https://github.com/simonw/datasette/issues/1356#issuecomment-853895159 https://api.github.com/repos/simonw/datasette/issues/1356 853895159 MDEyOklzc3VlQ29tbWVudDg1Mzg5NTE1OQ== 25778 2021-06-03T14:03:59Z 2021-06-03T14:03:59Z NONE (Putting thoughts here to keep the conversation in one place.) I think using datasette for this use-case is the right approach. I usually have both datasette and sqlite-utils installed in the same project, and that's where I'm trying out queries, so it probably makes the most sense to have datasette also manage the output (and maybe the input, too). It seems like both `--get` and `--query` could work better as subcommands, rather than options, if you're looking at building out a full CLI experience in datasette. It would give a cleaner separation in what you're trying to do and let each have its own dedicated options. So something like this: ```sh # run an arbitrary query datasette query covid.db "select * from ny_times_us_counties limit 1" --format yaml # run a canned query datasette get covid.db some-canned-query --format yaml ``` 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 910092577  
https://github.com/simonw/sqlite-utils/issues/264#issuecomment-853567861 https://api.github.com/repos/simonw/sqlite-utils/issues/264 853567861 MDEyOklzc3VlQ29tbWVudDg1MzU2Nzg2MQ== 9599 2021-06-03T05:12:21Z 2021-06-03T05:12:21Z OWNER I think this is more likely to happen in Datasette than in sqlite-utils - see https://github.com/simonw/datasette/issues/1356 for thoughts on this. 
{
    "total_count": 1,
    "+1": 1,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 907642546  
https://github.com/simonw/datasette/issues/526#issuecomment-853567413 https://api.github.com/repos/simonw/datasette/issues/526 853567413 MDEyOklzc3VlQ29tbWVudDg1MzU2NzQxMw== 9599 2021-06-03T05:11:27Z 2021-06-03T05:11:27Z OWNER Another potential way to implement this would be to hold the SQLite connection open and execute the full query there. I've avoided this in the past due to concerns of resource exhaustion - if multiple requests attempt this at the same time all of the connections in the pool will become tied up and the site will be unable to respond to further requests. But... now that Datasette has authentication there's the possibility of making this feature only available to specific authenticated users - the `--root` user for example. Which avoids the danger while unlocking a super-useful feature. Not to mention people who are running Datasette privately on their own laptop, or the proposed `--query` CLI feature in #1356. 
{
    "total_count": 1,
    "+1": 1,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 459882902  
https://github.com/simonw/datasette/issues/1356#issuecomment-853566337 https://api.github.com/repos/simonw/datasette/issues/1356 853566337 MDEyOklzc3VlQ29tbWVudDg1MzU2NjMzNw== 9599 2021-06-03T05:08:32Z 2021-06-03T05:08:32Z OWNER Also relevant: CSV streaming for canned queries in #526 - even better if we could stream ANY SQL query. 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 910092577  
https://github.com/simonw/datasette/issues/1062#issuecomment-853566011 https://api.github.com/repos/simonw/datasette/issues/1062 853566011 MDEyOklzc3VlQ29tbWVudDg1MzU2NjAxMQ== 9599 2021-06-03T05:07:42Z 2021-06-03T05:07:42Z OWNER Implementing this would make #1356 a whole lot more interesting. 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 732674148  
https://github.com/simonw/datasette/issues/1356#issuecomment-853565850 https://api.github.com/repos/simonw/datasette/issues/1356 853565850 MDEyOklzc3VlQ29tbWVudDg1MzU2NTg1MA== 9599 2021-06-03T05:07:21Z 2021-06-03T05:07:21Z OWNER A problem with this is that if you're using `--query` you likely want ALL of the results - at the moment the only Datasette output type that can stream everything is `.csv` and plugin formats can't handle full streams, see #1062 and #1177. So there's not much point implementing this unless we first make plugins able to add custom streaming formats. 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 910092577  
https://github.com/simonw/datasette/issues/1356#issuecomment-853562891 https://api.github.com/repos/simonw/datasette/issues/1356 853562891 MDEyOklzc3VlQ29tbWVudDg1MzU2Mjg5MQ== 9599 2021-06-03T04:59:40Z 2021-06-03T04:59:40Z OWNER It's weird that `--get` is documented on this page right now: https://docs.datasette.io/en/stable/getting_started.html#datasette-get If I implement this I should build a new "Datasette on the command-line" documentation page to cover both `--get` and `--query`. 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 910092577  
https://github.com/simonw/datasette/issues/1356#issuecomment-853560870 https://api.github.com/repos/simonw/datasette/issues/1356 853560870 MDEyOklzc3VlQ29tbWVudDg1MzU2MDg3MA== 9599 2021-06-03T04:53:47Z 2021-06-03T04:53:56Z OWNER This is also interesting when used in conjunction with the proposed `datasette insert` command from #1163 - Datasette would become a plugin-driven CLI tool for both ingesting and outputting data, as a side-effect of its web features. 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 910092577  
https://github.com/simonw/datasette/issues/1356#issuecomment-853560389 https://api.github.com/repos/simonw/datasette/issues/1356 853560389 MDEyOklzc3VlQ29tbWVudDg1MzU2MDM4OQ== 9599 2021-06-03T04:52:13Z 2021-06-03T04:52:13Z OWNER I should implement #1355 for more efficient `--csv` streaming as part of this. 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 910092577  
https://github.com/simonw/datasette/issues/1356#issuecomment-853560237 https://api.github.com/repos/simonw/datasette/issues/1356 853560237 MDEyOklzc3VlQ29tbWVudDg1MzU2MDIzNw== 9599 2021-06-03T04:51:49Z 2021-06-03T04:51:49Z OWNER This feels like a relatively simple feature to implement that unlocks a whole new set of possible uses for Datasette - as described by @eyeseast in https://github.com/simonw/sqlite-utils/issues/264#issue-907642546. 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 910092577  
https://github.com/simonw/datasette/issues/1356#issuecomment-853559915 https://api.github.com/repos/simonw/datasette/issues/1356 853559915 MDEyOklzc3VlQ29tbWVudDg1MzU1OTkxNQ== 9599 2021-06-03T04:50:52Z 2021-06-03T04:50:52Z OWNER What happens if you pass multiple databases? The `--query` would be executed against the first one. And if you pass `--crossdb` it would be executed against the `/_memory` database and would support cross-database joins. Key thing here is that output plugins are supported (also plugins that add new SQL functions), making many Datasette plugins usable from the command-line. 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 910092577  
https://github.com/simonw/sqlite-utils/issues/264#issuecomment-853558741 https://api.github.com/repos/simonw/sqlite-utils/issues/264 853558741 MDEyOklzc3VlQ29tbWVudDg1MzU1ODc0MQ== 9599 2021-06-03T04:47:19Z 2021-06-03T04:47:19Z OWNER This inspired me to re-examine how `--get` works, hence this issue: https://github.com/simonw/datasette/issues/1355 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 907642546  
https://github.com/simonw/datasette/issues/1355#issuecomment-853557439 https://api.github.com/repos/simonw/datasette/issues/1355 853557439 MDEyOklzc3VlQ29tbWVudDg1MzU1NzQzOQ== 9599 2021-06-03T04:43:14Z 2021-06-03T04:43:14Z OWNER It's using `TestClient` at the moment which is a wrapper around `httpx` (as of ) that uses the `@async_to_sync` decorator to hide the async nature. https://github.com/simonw/datasette/blob/f78ebdc04537a6102316d6dbbf6c887565806078/datasette/utils/testing.py#L102-L156 Maybe the fix here is to switch the `--get` implementation to using `httpx` directly with https://www.python-httpx.org/async/#streaming-responses 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 910088936  
https://github.com/simonw/sqlite-utils/issues/264#issuecomment-853553754 https://api.github.com/repos/simonw/sqlite-utils/issues/264 853553754 MDEyOklzc3VlQ29tbWVudDg1MzU1Mzc1NA== 9599 2021-06-03T04:32:42Z 2021-06-03T04:36:36Z OWNER This is a really interesting thought. I've so far resisted the temptation to add plugins to `sqlite-utils`, partly to avoid overlap with Datasette - but I'm open to discussing it. There's actually a way for you to do what you're describing using `datasette` on the command-line, though it's a little obscure - also Datasette doesn't yet have a GeoJSON output extension, though it really should have one. Here's an example using [datasette-yaml](https://datasette.io/plugins/datasette-yaml): ``` datasette /tmp/covid.db --get='/covid/ny_times_us_counties.yaml' - rowid: 1 date: '2020-01-21' county: Snohomish state: Washington fips: 53061 cases: 1 deaths: 0 - rowid: 2 date: '2020-01-22' county: Snohomish state: Washington fips: 53061 cases: 1 deaths: 0 ``` It even works with arbitrary SQL queries, though you might have to apply URL encoding to the `--get` string (this seems to work though): ``` datasette /tmp/covid.db --get='/covid.yaml?sql=select * from ny_times_us_counties limit 2' - date: '2020-01-21' county: Snohomish state: Washington fips: 53061 cases: 1 deaths: 0 - date: '2020-01-22' county: Snohomish state: Washington fips: 53061 cases: 1 deaths: 0 ``` Here's the documentation for `--get`: https://docs.datasette.io/en/latest/getting_started.html#datasette-get 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 907642546  
https://github.com/simonw/sqlite-utils/issues/263#issuecomment-853554550 https://api.github.com/repos/simonw/sqlite-utils/issues/263 853554550 MDEyOklzc3VlQ29tbWVudDg1MzU1NDU1MA== 9599 2021-06-03T04:34:38Z 2021-06-03T04:34:38Z OWNER Documentation: https://sqlite-utils.datasette.io/en/latest/cli.html#listing-indexes 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 906356331  
https://github.com/simonw/sqlite-utils/issues/263#issuecomment-853548442 https://api.github.com/repos/simonw/sqlite-utils/issues/263 853548442 MDEyOklzc3VlQ29tbWVudDg1MzU0ODQ0Mg== 9599 2021-06-03T04:16:49Z 2021-06-03T04:16:49Z OWNER Needs to show the table each index applies to. 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 906356331  
https://github.com/simonw/sqlite-utils/issues/263#issuecomment-853547681 https://api.github.com/repos/simonw/sqlite-utils/issues/263 853547681 MDEyOklzc3VlQ29tbWVudDg1MzU0NzY4MQ== 9599 2021-06-03T04:14:28Z 2021-06-03T04:14:28Z OWNER This looks good: ``` % sqlite-utils indexes /tmp/covid.db -t index_name seqno cid name desc coll key ------------------------------------------------------ ------- ----- ----------------- ------ ------ ----- idx_johns_hopkins_csse_daily_reports_combined_key 0 12 combined_key 0 BINARY 1 idx_johns_hopkins_csse_daily_reports_country_or_region 0 1 country_or_region 0 BINARY 1 idx_johns_hopkins_csse_daily_reports_province_or_state 0 2 province_or_state 0 BINARY 1 idx_johns_hopkins_csse_daily_reports_day 0 0 day 0 BINARY 1 idx_ny_times_us_counties_date 0 0 date 1 BINARY 1 idx_ny_times_us_counties_fips 0 3 fips 0 BINARY 1 idx_ny_times_us_counties_county 0 1 county 0 BINARY 1 idx_ny_times_us_counties_state 0 2 state 0 BINARY 1 % sqlite-utils indexes /tmp/covid.db -t --aux index_name seqno cid name desc coll key ------------------------------------------------------ ------- ----- ----------------- ------ ------ ----- idx_johns_hopkins_csse_daily_reports_combined_key 0 12 combined_key 0 BINARY 1 idx_johns_hopkins_csse_daily_reports_combined_key 1 -1 0 BINARY 0 idx_johns_hopkins_csse_daily_reports_country_or_region 0 1 country_or_region 0 BINARY 1 idx_johns_hopkins_csse_daily_reports_country_or_region 1 -1 0 BINARY 0 idx_johns_hopkins_csse_daily_reports_province_or_state 0 2 province_or_state 0 BINARY 1 idx_johns_hopkins_csse_daily_reports_province_or_state 1 -1 0 BINARY 0 idx_johns_hopkins_csse_daily_reports_day 0 0 day 0 BINARY 1 idx_johns_hopkins_csse_daily_reports_day 1 -1 0 BINARY 0 idx_ny_times_us_counties_date 0 0 date 1 BINARY 1 idx_ny_times_us_counties_date 1 -1 0 BINARY 0 idx_ny_times_us_counties_fips 0 3 fips 0 BINARY 1 idx_ny_times_us_counties_fips 1 -1 0 BINARY 0 idx_ny_times_us_counties_county 0 1 county 0 BINARY 1 idx_ny_times_us_counties_county 1 -1 0 BINARY 0 idx_ny_times_us_counties_state 0 2 state 0 BINARY 1 idx_ny_times_us_counties_state 1 -1 0 BINARY 0 ``` 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 906356331  
https://github.com/simonw/sqlite-utils/issues/263#issuecomment-853546818 https://api.github.com/repos/simonw/sqlite-utils/issues/263 853546818 MDEyOklzc3VlQ29tbWVudDg1MzU0NjgxOA== 9599 2021-06-03T04:11:46Z 2021-06-03T04:11:46Z OWNER By default I won't return auxiliary columns, but I'll offer a `--aux` option to return them. 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 906356331  
https://github.com/simonw/sqlite-utils/issues/263#issuecomment-853545743 https://api.github.com/repos/simonw/sqlite-utils/issues/263 853545743 MDEyOklzc3VlQ29tbWVudDg1MzU0NTc0Mw== 9599 2021-06-03T04:08:04Z 2021-06-03T04:08:04Z OWNER Figuring out the right queries: https://covid-19.datasettes.com/covid?sql=select+sqlite_master.name%2C+i.*+from+sqlite_master%0D%0Ajoin+pragma_index_list%28sqlite_master.name%29+i%0D%0Awhere+type+%3D+%27table%27 This query shows all columns across all indexes across all tables: ```sql select i.name as index_name, xinfo.* from sqlite_master join pragma_index_list(sqlite_master.name) i join pragma_index_xinfo(index_name) xinfo where sqlite_master.type = 'table' ``` https://covid-19.datasettes.com/covid?sql=select+i.name+as+index_name%2C+xinfo.*+from+sqlite_master%0D%0Ajoin+pragma_index_list%28sqlite_master.name%29+i%0D%0Ajoin+pragma_index_xinfo%28index_name%29+xinfo%0D%0Awhere+sqlite_master.type+%3D+%27table%27 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 906356331  
https://github.com/simonw/sqlite-utils/issues/263#issuecomment-853544493 https://api.github.com/repos/simonw/sqlite-utils/issues/263 853544493 MDEyOklzc3VlQ29tbWVudDg1MzU0NDQ5Mw== 9599 2021-06-03T04:03:59Z 2021-06-03T04:03:59Z OWNER Here's how `sqlite-utils triggers` works: https://github.com/simonw/sqlite-utils/blob/9c67cb925253cd5ef54a1fe0496e0ff9caeacfd6/sqlite_utils/cli.py#L1266-L1277 Running it from a SQL query makes it easy to support modifiers like `--csv` and `-t`. 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 906356331  
https://github.com/simonw/sqlite-utils/issues/261#issuecomment-853541869 https://api.github.com/repos/simonw/sqlite-utils/issues/261 853541869 MDEyOklzc3VlQ29tbWVudDg1MzU0MTg2OQ== 9599 2021-06-03T03:54:14Z 2021-06-03T03:54:14Z OWNER Documentation: https://sqlite-utils.datasette.io/en/latest/python-api.html#xindexes 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 906345899  
https://github.com/simonw/sqlite-utils/issues/261#issuecomment-853535559 https://api.github.com/repos/simonw/sqlite-utils/issues/261 853535559 MDEyOklzc3VlQ29tbWVudDg1MzUzNTU1OQ== 9599 2021-06-03T03:32:47Z 2021-06-03T03:32:47Z OWNER New design: ```python def test_xindexes(fresh_db): fresh_db.executescript( """ create table Gosh (c1 text, c2 text, c3 text); create index Gosh_c1 on Gosh(c1); create index Gosh_c2c3 on Gosh(c2, c3 desc); """ ) assert fresh_db["Gosh"].xindexes == [ XIndex( name="Gosh_c2c3", columns=[ XIndexColumn(seqno=0, cid=1, name="c2", desc=0, coll="BINARY", key=1), XIndexColumn(seqno=1, cid=2, name="c3", desc=1, coll="BINARY", key=1), XIndexColumn(seqno=2, cid=-1, name=None, desc=0, coll="BINARY", key=0), ], ), XIndex( name="Gosh_c1", columns=[ XIndexColumn(seqno=0, cid=0, name="c1", desc=0, coll="BINARY", key=1), XIndexColumn(seqno=1, cid=-1, name=None, desc=0, coll="BINARY", key=0), ], ), ] ``` 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 906345899  
https://github.com/simonw/sqlite-utils/issues/261#issuecomment-853534732 https://api.github.com/repos/simonw/sqlite-utils/issues/261 853534732 MDEyOklzc3VlQ29tbWVudDg1MzUzNDczMg== 9599 2021-06-03T03:30:10Z 2021-06-03T03:30:10Z OWNER I'm going to return `XIndex(name, columns)` - where `columns` is a list of `XIndexColumn`. 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 906345899  
https://github.com/simonw/sqlite-utils/issues/261#issuecomment-853530348 https://api.github.com/repos/simonw/sqlite-utils/issues/261 853530348 MDEyOklzc3VlQ29tbWVudDg1MzUzMDM0OA== 9599 2021-06-03T03:16:33Z 2021-06-03T03:16:33Z OWNER In prototyping this out I realize that I actually want to get back the name of each index, then for each of them the detailed list of index columns. Here's the test from my initial prototype: ```python def test_xindexes(fresh_db): fresh_db.executescript( """ create table Gosh (c1 text, c2 text, c3 text); create index Gosh_c1 on Gosh(c1); create index Gosh_c2c3 on Gosh(c2, c3 desc); """ ) assert fresh_db["Gosh"].xindexes == [ ( "Gosh_c2c3", [ XIndex(seqno=0, cid=1, name="c2", desc=0, coll="BINARY", key=1), XIndex(seqno=1, cid=2, name="c3", desc=1, coll="BINARY", key=1), XIndex(seqno=2, cid=-1, name=None, desc=0, coll="BINARY", key=0), ], ), ( "Gosh_c1", [ XIndex(seqno=0, cid=0, name="c1", desc=0, coll="BINARY", key=1), XIndex(seqno=1, cid=-1, name=None, desc=0, coll="BINARY", key=0), ], ), ] ``` 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 906345899  
https://github.com/simonw/sqlite-utils/issues/261#issuecomment-853525036 https://api.github.com/repos/simonw/sqlite-utils/issues/261 853525036 MDEyOklzc3VlQ29tbWVudDg1MzUyNTAzNg== 9599 2021-06-03T03:02:22Z 2021-06-03T03:02:22Z OWNER This would be a breaking change - and the fact that it returns auxiliary columns isn't particularly useful for most cases - "Auxiliary columns are additional columns needed to locate the table entry that corresponds to each index entry". Instead, I'm going to add a new property `table.xindexes` which exposes this. 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 906345899  
https://github.com/simonw/datasette/issues/893#issuecomment-852712469 https://api.github.com/repos/simonw/datasette/issues/893 852712469 MDEyOklzc3VlQ29tbWVudDg1MjcxMjQ2OQ== 9599 2021-06-02T04:29:56Z 2021-06-02T04:29:56Z OWNER I think I fixed this a while ago, though I can't find the commit now. Please re-open if the bug still happens! 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 656959584  
https://github.com/simonw/datasette/issues/1127#issuecomment-852712106 https://api.github.com/repos/simonw/datasette/issues/1127 852712106 MDEyOklzc3VlQ29tbWVudDg1MjcxMjEwNg== 9599 2021-06-02T04:28:55Z 2021-06-02T04:28:55Z OWNER This became resizable in #1236. 
{
    "total_count": 1,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 1,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 756818250  
https://github.com/simonw/datasette/pull/1306#issuecomment-852710117 https://api.github.com/repos/simonw/datasette/issues/1306 852710117 MDEyOklzc3VlQ29tbWVudDg1MjcxMDExNw== 9599 2021-06-02T04:24:33Z 2021-06-02T04:24:33Z OWNER I have a test - I'll merge this and then add that test. 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 864979486  
https://github.com/simonw/datasette/pull/1306#issuecomment-852708574 https://api.github.com/repos/simonw/datasette/issues/1306 852708574 MDEyOklzc3VlQ29tbWVudDg1MjcwODU3NA== 9599 2021-06-02T04:20:18Z 2021-06-02T04:20:18Z OWNER https://github.com/simonw/datasette/blob/115332ce76c0e867d9936406aaf4bcee6b1ef3cb/datasette/views/index.py#L77 helps here - looks like we can pass `?_sort=relationships` to trigger the bug. 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 864979486  
https://github.com/simonw/datasette/pull/1306#issuecomment-852707676 https://api.github.com/repos/simonw/datasette/issues/1306 852707676 MDEyOklzc3VlQ29tbWVudDg1MjcwNzY3Ng== 9599 2021-06-02T04:17:41Z 2021-06-02T04:17:41Z OWNER We need to add a test that illustrates the bug in #1305 so we can prove that this definitely fixes it. 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 864979486  
https://github.com/simonw/datasette/issues/619#issuecomment-852706970 https://api.github.com/repos/simonw/datasette/issues/619 852706970 MDEyOklzc3VlQ29tbWVudDg1MjcwNjk3MA== 9599 2021-06-02T04:15:54Z 2021-06-02T04:15:54Z OWNER <img width="1009" alt="fixtures__select___from__foo__and_Facets_should_not_execute_for__shape_array_object_·_Issue__263_·_simonw_datasette" src="https://user-images.githubusercontent.com/9599/120423073-8a338700-c31e-11eb-8c8f-affc24708b32.png"> 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 520655983  
https://github.com/simonw/datasette/issues/619#issuecomment-852703357 https://api.github.com/repos/simonw/datasette/issues/619 852703357 MDEyOklzc3VlQ29tbWVudDg1MjcwMzM1Nw== 9599 2021-06-02T04:08:03Z 2021-06-02T04:08:03Z OWNER A SQL error now looks like this: https://latest.datasette.io/fixtures?sql=select%0D%0A++*%0D%0Afrom%0D%0A++%5Bfoo%5D <img width="1023" alt="fixtures__select___from__foo_" src="https://user-images.githubusercontent.com/9599/120422579-691e6680-c31d-11eb-99ea-655bdc2a401d.png"> I'm going to get rid of that "0 results" message if an error is shown. 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 520655983  
https://github.com/simonw/datasette/pull/1346#issuecomment-852693854 https://api.github.com/repos/simonw/datasette/issues/1346 852693854 MDEyOklzc3VlQ29tbWVudDg1MjY5Mzg1NA== 22429695 2021-06-02T03:44:35Z 2021-06-02T03:44:35Z NONE # [Codecov](https://codecov.io/gh/simonw/datasette/pull/1346?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) Report > Merging [#1346](https://codecov.io/gh/simonw/datasette/pull/1346?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (3bffc35) into [main](https://codecov.io/gh/simonw/datasette/commit/7b106e106000713bbee31b34d694b3dadbd4818c?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) (7b106e1) will **increase** coverage by `0.14%`. > The diff coverage is `100.00%`. [![Impacted file tree graph](https://codecov.io/gh/simonw/datasette/pull/1346/graphs/tree.svg?width=650&height=150&src=pr&token=eSahVY7kw1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison)](https://codecov.io/gh/simonw/datasette/pull/1346?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) ```diff @@ Coverage Diff @@ ## main #1346 +/- ## ========================================== + Coverage 91.56% 91.71% +0.14% ========================================== Files 34 34 Lines 4282 4332 +50 ========================================== + Hits 3921 3973 +52 + Misses 361 359 -2 ``` | [Impacted Files](https://codecov.io/gh/simonw/datasette/pull/1346?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) | Coverage Δ | | |---|---|---| | [datasette/app.py](https://codecov.io/gh/simonw/datasette/pull/1346/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL2FwcC5weQ==) | `95.69% <ø> (-0.15%)` | :arrow_down: | | [datasette/renderer.py](https://codecov.io/gh/simonw/datasette/pull/1346/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3JlbmRlcmVyLnB5) | `94.20% <100.00%> (+0.17%)` | :arrow_up: | | [datasette/utils/\_\_init\_\_.py](https://codecov.io/gh/simonw/datasette/pull/1346/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3V0aWxzL19faW5pdF9fLnB5) | `94.36% <100.00%> (+0.05%)` | :arrow_up: | | [datasette/views/base.py](https://codecov.io/gh/simonw/datasette/pull/1346/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL2Jhc2UucHk=) | `95.41% <100.00%> (+0.39%)` | :arrow_up: | | [datasette/views/database.py](https://codecov.io/gh/simonw/datasette/pull/1346/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL2RhdGFiYXNlLnB5) | `97.28% <100.00%> (+0.09%)` | :arrow_up: | | [datasette/views/table.py](https://codecov.io/gh/simonw/datasette/pull/1346/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3ZpZXdzL3RhYmxlLnB5) | `95.90% <100.00%> (+0.07%)` | :arrow_up: | | [datasette/tracer.py](https://codecov.io/gh/simonw/datasette/pull/1346/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison#diff-ZGF0YXNldHRlL3RyYWNlci5weQ==) | `85.05% <0.00%> (+3.44%)` | :arrow_up: | ------ [Continue to review full report at Codecov](https://codecov.io/gh/simonw/datasette/pull/1346?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison) > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/simonw/datasette/pull/1346?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Last update [7b106e1...3bffc35](https://codecov.io/gh/simonw/datasette/pull/1346?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Simon+Willison). 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 904537568  
https://github.com/simonw/datasette/issues/1257#issuecomment-852686827 https://api.github.com/repos/simonw/datasette/issues/1257 852686827 MDEyOklzc3VlQ29tbWVudDg1MjY4NjgyNw== 9599 2021-06-02T03:26:27Z 2021-06-02T03:26:27Z OWNER I tried and failed to get this fix working for tables with double quotes in their name - I couldn't figure out what the double-quote-in-a-table-name version of this code would look like: https://github.com/simonw/datasette/blob/807de378d08752a0f05bb1b980a0a62620a70520/datasette/utils/__init__.py#L538-L548 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 828811618  
https://github.com/simonw/datasette/issues/1257#issuecomment-852681622 https://api.github.com/repos/simonw/datasette/issues/1257 852681622 MDEyOklzc3VlQ29tbWVudDg1MjY4MTYyMg== 9599 2021-06-02T03:12:18Z 2021-06-02T03:12:18Z OWNER Created a test database like this: ``` % sqlite-utils create-table quote-in-name.db "this'hasquoteinname" id integer name text --pk id % datasette quote-in-name.db -p 8025 --pdb INFO: Started server process [86046] INFO: Waiting for application startup. INFO: Application startup complete. INFO: Uvicorn running on http://127.0.0.1:8025 (Press CTRL+C to quit) > /Users/simon/Dropbox/Development/datasette/datasette/utils/__init__.py(530)detect_fts() -> rows = conn.execute(detect_fts_sql(table)).fetchall() (Pdb) c Traceback (most recent call last): File "/Users/simon/Dropbox/Development/datasette/datasette/app.py", line 1124, in route_path response = await view(request, send) File "/Users/simon/Dropbox/Development/datasette/datasette/views/base.py", line 147, in view return await self.dispatch_request( File "/Users/simon/Dropbox/Development/datasette/datasette/views/base.py", line 122, in dispatch_request return await handler(request, *args, **kwargs) File "/Users/simon/Dropbox/Development/datasette/datasette/views/index.py", line 72, in get "fts_table": await db.fts_table(table), File "/Users/simon/Dropbox/Development/datasette/datasette/database.py", line 279, in fts_table return await self.execute_fn(lambda conn: detect_fts(conn, table)) File "/Users/simon/Dropbox/Development/datasette/datasette/database.py", line 155, in execute_fn return await asyncio.get_event_loop().run_in_executor( File "/Users/simon/.pyenv/versions/3.8.2/lib/python3.8/concurrent/futures/thread.py", line 57, in run result = self.fn(*self.args, **self.kwargs) File "/Users/simon/Dropbox/Development/datasette/datasette/database.py", line 153, in in_thread return fn(conn) File "/Users/simon/Dropbox/Development/datasette/datasette/database.py", line 279, in <lambda> return await self.execute_fn(lambda conn: detect_fts(conn, table)) File "/Users/simon/Dropbox/Development/datasette/datasette/utils/__init__.py", line 530, in detect_fts rows = conn.execute(detect_fts_sql(table)).fetchall() sqlite3.OperationalError: near "hasquoteinname": syntax error ``` 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 828811618  
https://github.com/simonw/datasette/pull/1352#issuecomment-852673695 https://api.github.com/repos/simonw/datasette/issues/1352 852673695 MDEyOklzc3VlQ29tbWVudDg1MjY3MzY5NQ== 9599 2021-06-02T02:52:26Z 2021-06-02T02:52:26Z OWNER @dependabot recreate 
{
    "total_count": 1,
    "+1": 1,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 908276134  
https://github.com/simonw/datasette/issues/263#issuecomment-852256784 https://api.github.com/repos/simonw/datasette/issues/263 852256784 MDEyOklzc3VlQ29tbWVudDg1MjI1Njc4NA== 9599 2021-06-01T16:20:09Z 2021-06-01T16:20:09Z OWNER This is a lot easier to implement now that we have a `?_nofacet=1` option from #1350. 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 323671577  
https://github.com/simonw/datasette/issues/1350#issuecomment-852256454 https://api.github.com/repos/simonw/datasette/issues/1350 852256454 MDEyOklzc3VlQ29tbWVudDg1MjI1NjQ1NA== 9599 2021-06-01T16:19:38Z 2021-06-01T16:19:38Z OWNER I renamed this to `?_nofacet=1` in #1353. 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 906977719  
https://github.com/simonw/datasette/issues/1353#issuecomment-852255951 https://api.github.com/repos/simonw/datasette/issues/1353 852255951 MDEyOklzc3VlQ29tbWVudDg1MjI1NTk1MQ== 9599 2021-06-01T16:18:54Z 2021-06-01T16:18:54Z OWNER Documented here: https://docs.datasette.io/en/latest/json_api.html#special-table-arguments 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 908446997  
https://github.com/simonw/datasette/issues/1354#issuecomment-852254298 https://api.github.com/repos/simonw/datasette/issues/1354 852254298 MDEyOklzc3VlQ29tbWVudDg1MjI1NDI5OA== 9599 2021-06-01T16:16:32Z 2021-06-01T16:16:32Z OWNER Running `python update-docs-help.py` helps fix this. 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 908465747  
https://github.com/simonw/datasette/issues/1353#issuecomment-852238201 https://api.github.com/repos/simonw/datasette/issues/1353 852238201 MDEyOklzc3VlQ29tbWVudDg1MjIzODIwMQ== 9599 2021-06-01T15:53:57Z 2021-06-01T15:53:57Z OWNER I'm going to rename `?_nofacets=1` to `?_nofacet=1` to keep it consistent with the new `?_nocount=1` option (and because I don't like `?_nocounts=1`). 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 908446997  
https://github.com/simonw/datasette/issues/1349#issuecomment-852237347 https://api.github.com/repos/simonw/datasette/issues/1349 852237347 MDEyOklzc3VlQ29tbWVudDg1MjIzNzM0Nw== 9599 2021-06-01T15:52:50Z 2021-06-01T15:52:50Z OWNER Fixed in https://github.com/simonw/datasette/commit/d1d06ace49606da790a765689b4fbffa4c6deecb 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 906385991  
https://github.com/simonw/datasette/issues/1284#issuecomment-851567204 https://api.github.com/repos/simonw/datasette/issues/1284 851567204 MDEyOklzc3VlQ29tbWVudDg1MTU2NzIwNA== 192568 2021-05-31T15:42:10Z 2021-05-31T15:42:10Z CONTRIBUTOR I very much want to make: https://disinfectants.SaferOrToxic.org/disinfectants/listN have this URL: https://disinfectants.SaferOrToxic.org/ I'm using only one table page on the site, with no pagination. I'm not using the home page, though when I tried to move my table to the home page as mentioned above, I failed to figure out how. I am using cloudflare, but I haven't figured out a forwarding or HTML re-write method of doing this, either. Is there any way I can get a prettier list URL? I'm on Vercel. (I have a wordpress site on the main domain on a separate host.) 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 845794436  
https://github.com/simonw/datasette/issues/1351#issuecomment-851133471 https://api.github.com/repos/simonw/datasette/issues/1351 851133471 MDEyOklzc3VlQ29tbWVudDg1MTEzMzQ3MQ== 9599 2021-05-31T03:02:59Z 2021-05-31T03:02:59Z OWNER Since traces only work with `text/html` and JSON at the moment, the easiest way to do this will be to wrap generated CSV in a HTML page in a textarea if the user specified `?_trace=1`: https://github.com/simonw/datasette/blob/c5ae1197a208e1b034c88882e3ac865813a40980/datasette/tracer.py#L125-L134 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 906993731  
https://github.com/simonw/datasette/issues/1349#issuecomment-851133125 https://api.github.com/repos/simonw/datasette/issues/1349 851133125 MDEyOklzc3VlQ29tbWVudDg1MTEzMzEyNQ== 9599 2021-05-31T03:01:48Z 2021-05-31T03:01:48Z OWNER I think it's worth getting `?_trace=1` to work with streaming CSV - this would have helped me spot this issue a long time ago. 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 906385991  
https://github.com/simonw/datasette/issues/1349#issuecomment-851131999 https://api.github.com/repos/simonw/datasette/issues/1349 851131999 MDEyOklzc3VlQ29tbWVudDg1MTEzMTk5OQ== 9599 2021-05-31T02:57:25Z 2021-05-31T02:57:25Z OWNER I'm having a really hard time figuring out how to unit test this - ideally I'd monitor which SQL queries are executed using the tracing mechanism, but that's not set up to work with anything other than HTML or JSON outputs: https://github.com/simonw/datasette/blob/c5ae1197a208e1b034c88882e3ac865813a40980/datasette/tracer.py#L125-L134 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
 906385991