5,814 rows sorted by updated_at descending

View and edit SQL

author_association

id html_url issue_url node_id user created_at updated_at ▲ author_association body reactions issue performed_via_github_app
851113886 https://github.com/simonw/datasette/issues/1345#issuecomment-851113886 https://api.github.com/repos/simonw/datasette/issues/1345 MDEyOklzc3VlQ29tbWVudDg1MTExMzg4Ng== simonw 9599 2021-05-31T01:52:27Z 2021-05-31T01:52:27Z OWNER

Related issue: visit https://latest.datasette.io/fixtures/facetable?_facet=state and click "Hide this column" on the "state" cog menu and you get https://latest.datasette.io/fixtures/facetable?_facet=state&_nocol=state which shows an error:

Invalid SQL

no such column: state

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
?_nocol= does not interact well with default facets 904071938  
850778311 https://github.com/simonw/datasette/issues/1349#issuecomment-850778311 https://api.github.com/repos/simonw/datasette/issues/1349 MDEyOklzc3VlQ29tbWVudDg1MDc3ODMxMQ== simonw 9599 2021-05-29T06:12:21Z 2021-05-29T06:12:21Z OWNER

It's not just facets, I think it's trying to execute suggested facets too!

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
CSV ?_stream=on redundantly calculates facets for every page 906385991  
850778273 https://github.com/simonw/datasette/issues/1349#issuecomment-850778273 https://api.github.com/repos/simonw/datasette/issues/1349 MDEyOklzc3VlQ29tbWVudDg1MDc3ODI3Mw== simonw 9599 2021-05-29T06:11:59Z 2021-05-29T06:11:59Z OWNER

Related issue: https://github.com/simonw/datasette/issues/263 - "Facets should not execute for ?shape=array|object"

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
CSV ?_stream=on redundantly calculates facets for every page 906385991  
850778230 https://github.com/simonw/datasette/issues/1349#issuecomment-850778230 https://api.github.com/repos/simonw/datasette/issues/1349 MDEyOklzc3VlQ29tbWVudDg1MDc3ODIzMA== simonw 9599 2021-05-29T06:11:28Z 2021-05-29T06:11:28Z OWNER
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
CSV ?_stream=on redundantly calculates facets for every page 906385991  
850772428 https://github.com/simonw/sqlite-utils/issues/250#issuecomment-850772428 https://api.github.com/repos/simonw/sqlite-utils/issues/250 MDEyOklzc3VlQ29tbWVudDg1MDc3MjQyOA== simonw 9599 2021-05-29T05:16:02Z 2021-05-29T05:28:57Z OWNER

I needed to find some CSV files on my computer with a BOM at the beginning - I figured out this recipe:

% rg -U -E none '^(?-u:\xEF\xBB\xBF)' --glob '*.csv' .

TIL here: https://til.simonwillison.net/bash/finding-bom-csv-files-with-ripgrep

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Handle byte order marks (BOMs) in CSV files 838148087  
850771264 https://github.com/simonw/sqlite-utils/issues/250#issuecomment-850771264 https://api.github.com/repos/simonw/sqlite-utils/issues/250 MDEyOklzc3VlQ29tbWVudDg1MDc3MTI2NA== simonw 9599 2021-05-29T05:06:13Z 2021-05-29T05:06:13Z OWNER

The other option is to check if the file starts with codecs.BOM_UTF8 - which is b'\xef\xbb\xbf'.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Handle byte order marks (BOMs) in CSV files 838148087  
850771017 https://github.com/simonw/sqlite-utils/issues/250#issuecomment-850771017 https://api.github.com/repos/simonw/sqlite-utils/issues/250 MDEyOklzc3VlQ29tbWVudDg1MDc3MTAxNw== simonw 9599 2021-05-29T05:04:28Z 2021-05-29T05:04:28Z OWNER

https://stackoverflow.com/a/44573867/6083 says:

There is no reason to check if a BOM exists or not, utf-8-sig manages that for you and behaves exactly as utf-8 if the BOM does not exist

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Handle byte order marks (BOMs) in CSV files 838148087  
850769067 https://github.com/simonw/sqlite-utils/issues/260#issuecomment-850769067 https://api.github.com/repos/simonw/sqlite-utils/issues/260 MDEyOklzc3VlQ29tbWVudDg1MDc2OTA2Nw== simonw 9599 2021-05-29T04:48:10Z 2021-05-29T04:48:10Z OWNER

I confirmed and it's possible to have a SQLite column with a hyphen at the start, confirmed using:

% sqlite-utils create-table demo.db demo -- id integer name text -blah integer
% sqlite-utils tables --schema demo.db -t
table    schema
-------  ---------------------
demo     CREATE TABLE [demo] (
            [id] INTEGER,
            [name] TEXT,
            [-blah] INTEGER
         )
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Support creating descending order indexes 906330187  
850768315 https://github.com/simonw/sqlite-utils/issues/260#issuecomment-850768315 https://api.github.com/repos/simonw/sqlite-utils/issues/260 MDEyOklzc3VlQ29tbWVudDg1MDc2ODMxNQ== simonw 9599 2021-05-29T04:39:33Z 2021-05-29T04:39:33Z OWNER

This doesn't work:

sqlite-utils create-index mydb.db dogs "-age" name

But this does:

sqlite-utils create-index mydb.db dogs -- -age name
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Support creating descending order indexes 906330187  
850767210 https://github.com/simonw/sqlite-utils/issues/260#issuecomment-850767210 https://api.github.com/repos/simonw/sqlite-utils/issues/260 MDEyOklzc3VlQ29tbWVudDg1MDc2NzIxMA== simonw 9599 2021-05-29T04:26:26Z 2021-05-29T04:28:31Z OWNER

It's weird having to use Database.DescIndex - I'm going to put DescIndex in sqlite_utils.db directly and let people import it.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Support creating descending order indexes 906330187  
850766552 https://github.com/simonw/sqlite-utils/issues/260#issuecomment-850766552 https://api.github.com/repos/simonw/sqlite-utils/issues/260 MDEyOklzc3VlQ29tbWVudDg1MDc2NjU1Mg== simonw 9599 2021-05-29T04:20:40Z 2021-05-29T04:24:01Z OWNER

PRAGMA index_xinfo(table) DOES return that data:

(Pdb) [c[0] for c in fresh_db.execute("PRAGMA index_xinfo('idx_dogs_age_name')").description]
['seqno', 'cid', 'name', 'desc', 'coll', 'key']
(Pdb) fresh_db.execute("PRAGMA index_xinfo('idx_dogs_age_name')").fetchall()
[(0, 2, 'age', 1, 'BINARY', 1), (1, 0, 'name', 0, 'BINARY', 1), (2, -1, None, 0, 'BINARY', 0)]

See https://sqlite.org/pragma.html#pragma_index_xinfo

Example output: https://covid-19.datasettes.com/covid?sql=select+*+from+pragma_index_xinfo%28%27idx_ny_times_us_counties_date%27%29

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Support creating descending order indexes 906330187  
850766335 https://github.com/simonw/sqlite-utils/issues/260#issuecomment-850766335 https://api.github.com/repos/simonw/sqlite-utils/issues/260 MDEyOklzc3VlQ29tbWVudDg1MDc2NjMzNQ== simonw 9599 2021-05-29T04:18:19Z 2021-05-29T04:18:19Z OWNER

Annoyingly the table.indexes property won't indicate if an index is in regular or reverse order - because the SQLite PRAGMA index_info(table) statement doesn't indicate that either. You have to look at the sqlite_master index definition to tell if any of the columns are in reverse order:

(Pdb) fresh_db.execute("select * from sqlite_master where type = 'index'").fetchall()
[('index', 'idx_dogs_age_name', 'dogs', 3, 'CREATE INDEX [idx_dogs_age_name]\n    ON [dogs] ([age] desc, [name])')]
(Pdb) fresh_db.execute("PRAGMA index_info('idx_dogs_age_name')").fetchall()
[(0, 2, 'age'), (1, 0, 'name')]
(Pdb) fresh_db.execute("PRAGMA index_info('idx_dogs_age_name')").description
(('seqno', None, None, None, None, None, None), ('cid', None, None, None, None, None, None), ('name', None, None, None, None, None, None))
(Pdb) dogs.indexes
[Index(seq=0, name='idx_dogs_age_name', unique=0, origin='c', partial=0, columns=['age', 'name'])]
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Support creating descending order indexes 906330187  
850765450 https://github.com/simonw/sqlite-utils/issues/260#issuecomment-850765450 https://api.github.com/repos/simonw/sqlite-utils/issues/260 MDEyOklzc3VlQ29tbWVudDg1MDc2NTQ1MA== simonw 9599 2021-05-29T04:09:13Z 2021-05-29T04:09:13Z OWNER

Decisions: for the Python API I'm going with db.DescIndex("column") as the way to do this.

For the CLI I'm going to do the "-age" thing.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Support creating descending order indexes 906330187  
850765291 https://github.com/simonw/sqlite-utils/issues/260#issuecomment-850765291 https://api.github.com/repos/simonw/sqlite-utils/issues/260 MDEyOklzc3VlQ29tbWVudDg1MDc2NTI5MQ== simonw 9599 2021-05-29T04:07:48Z 2021-05-29T04:08:21Z OWNER

For the CLI version I could say that you can use a - prefix to specify reverse direction:

sqlite-utils create-index mydb.db dogs -age name

No, that doesn't work - it could get confused with a command-line flag. I guess you could do this:

sqlite-utils create-index mydb.db dogs "-age" name

This does mean that if any of your column names begin with a hyphen you can't use the CLI to add indexes to them. Is that an acceptable limitation? Users can always use sqlite-utils mydb.db "create index ..." in that case.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Support creating descending order indexes 906330187  
850765050 https://github.com/simonw/sqlite-utils/issues/260#issuecomment-850765050 https://api.github.com/repos/simonw/sqlite-utils/issues/260 MDEyOklzc3VlQ29tbWVudDg1MDc2NTA1MA== simonw 9599 2021-05-29T04:05:24Z 2021-05-29T04:05:40Z OWNER

Need to solve this for the CLI tool too. Currently that works like this: https://sqlite-utils.datasette.io/en/stable/cli.html#creating-indexes

sqlite-utils create-index mydb.db mytable col1 [col2...]

Even harder to decide how to add a descending option to this. Maybe like this?

sqlite-utils create-index mydb.db mytable --direction col1 asc --direction col2 desc

It's a bit gross though! We're saying here that if a single one of the columns you are creating an index for is in reverse direction you have to use --direction to specify each end every other index.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Support creating descending order indexes 906330187  
850764700 https://github.com/simonw/sqlite-utils/issues/260#issuecomment-850764700 https://api.github.com/repos/simonw/sqlite-utils/issues/260 MDEyOklzc3VlQ29tbWVudDg1MDc2NDcwMA== simonw 9599 2021-05-29T04:02:10Z 2021-05-29T04:02:10Z OWNER

I could use db.desc_index("age") to match SQLite SQL syntax, which uses desc and not descending.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Support creating descending order indexes 906330187  
850764655 https://github.com/simonw/sqlite-utils/issues/260#issuecomment-850764655 https://api.github.com/repos/simonw/sqlite-utils/issues/260 MDEyOklzc3VlQ29tbWVudDg1MDc2NDY1NQ== simonw 9599 2021-05-29T04:01:41Z 2021-05-29T04:01:41Z OWNER

Maybe:

db["dogs"].create_index([db.descending_index("age"), "name"])

It's a little verbose but it's for a relatively rare activity and it does make it very clear what is going on.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Support creating descending order indexes 906330187  
850764594 https://github.com/simonw/sqlite-utils/issues/260#issuecomment-850764594 https://api.github.com/repos/simonw/sqlite-utils/issues/260 MDEyOklzc3VlQ29tbWVudDg1MDc2NDU5NA== simonw 9599 2021-05-29T04:00:54Z 2021-05-29T04:00:54Z OWNER

A few options:

db["dogs"].create_index([("age", "desc"), "name"])

db["dogs"].create_index([desc("age"), "name"])

db["dogs"].create_index([db.desc("age"), "name"])

The first option uses an optional tuple. The second two use a desc() function - the question is where should that live?

sqlite_utils.desc(column) or db.desc(column) are both options.

I don't like using the term desc() for "descending index" though - it feels like it should mean something more broad.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Support creating descending order indexes 906330187  
850764253 https://github.com/simonw/sqlite-utils/issues/260#issuecomment-850764253 https://api.github.com/repos/simonw/sqlite-utils/issues/260 MDEyOklzc3VlQ29tbWVudDg1MDc2NDI1Mw== simonw 9599 2021-05-29T03:57:54Z 2021-05-29T03:57:54Z OWNER

The problem here is differentiating between a column with the name date desc and wanting to create a descending index on a column called date.

This won't work:

db["ny_times_us_counties"].create_index(["date desc"], desc=True)

Because we need to be able to create compound indexes with columns with different directions - for example:

create index idx_age_desc_name on dogs (age desc, name)
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Support creating descending order indexes 906330187  
850583584 https://github.com/simonw/datasette/pull/1296#issuecomment-850583584 https://api.github.com/repos/simonw/datasette/issues/1296 MDEyOklzc3VlQ29tbWVudDg1MDU4MzU4NA== simonw 9599 2021-05-28T18:06:11Z 2021-05-28T18:06:11Z OWNER

As a bonus, the Docker image becomes smaller

That's a huge surprise to me! And most welcome.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Dockerfile: use Ubuntu 20.10 as base 855446829  
850077484 https://github.com/simonw/datasette/pull/1348#issuecomment-850077484 https://api.github.com/repos/simonw/datasette/issues/1348 MDEyOklzc3VlQ29tbWVudDg1MDA3NzQ4NA== codecov[bot] 22429695 2021-05-28T03:06:16Z 2021-05-28T03:06:16Z NONE

Codecov Report

Merging #1348 (56cba8f) into main (7b106e1) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main    #1348   +/-   ##
=======================================
  Coverage   91.56%   91.56%           
=======================================
  Files          34       34           
  Lines        4282     4282           
=======================================
  Hits         3921     3921           
  Misses        361      361           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7b106e1...56cba8f. Read the comment docs.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
DRAFT: add test and scan for docker images 904598267  
850077261 https://github.com/simonw/datasette/pull/1348#issuecomment-850077261 https://api.github.com/repos/simonw/datasette/issues/1348 MDEyOklzc3VlQ29tbWVudDg1MDA3NzI2MQ== blairdrummond 10801138 2021-05-28T03:05:38Z 2021-05-28T03:05:38Z CONTRIBUTOR

Note, the CVEs are probably resolvable with this https://github.com/simonw/datasette/pull/1296 . My experience is that Ubuntu seems to manage these better? Though that is surprising :/

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
DRAFT: add test and scan for docker images 904598267  
850059328 https://github.com/simonw/datasette/pull/1346#issuecomment-850059328 https://api.github.com/repos/simonw/datasette/issues/1346 MDEyOklzc3VlQ29tbWVudDg1MDA1OTMyOA== simonw 9599 2021-05-28T02:08:10Z 2021-05-28T02:09:25Z OWNER

This is the current test failure, but it actually another problem that we don't have tests in place for errors with different formats and shapes:

    def test_magic_parameters_cannot_be_used_in_arbitrary_queries(magic_parameters_client):
        response = magic_parameters_client.get(
            "/data.json?sql=select+:_header_host&_shape=array"
        )
        assert 400 == response.status
>       assert "You did not supply a value for binding 1." == response.json["error"]
E       TypeError: list indices must be integers or slices, not str

The test fails because response.json here is the empty list [].

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Re-display user's query with an error message if an error occurs 904537568  
850059607 https://github.com/simonw/datasette/issues/619#issuecomment-850059607 https://api.github.com/repos/simonw/datasette/issues/619 MDEyOklzc3VlQ29tbWVudDg1MDA1OTYwNw== simonw 9599 2021-05-28T02:08:58Z 2021-05-28T02:08:58Z OWNER

Remaining work will happen in #1346

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
"Invalid SQL" page should let you edit the SQL 520655983  
850058851 https://github.com/simonw/datasette/pull/1346#issuecomment-850058851 https://api.github.com/repos/simonw/datasette/issues/1346 MDEyOklzc3VlQ29tbWVudDg1MDA1ODg1MQ== simonw 9599 2021-05-28T02:06:46Z 2021-05-28T02:06:46Z OWNER

The one test failure here actually illustrates a larger problem: if the user specifies .json?_shape=array but an error occurs, what should we do?

Prior to this change we return the following JSON:

{
    "ok": false,
    "error": "You did not supply a value for binding 1.",
    "status": 500,
    "title": null
}

But this comes from the handle_500 higher level code here: https://github.com/simonw/datasette/blob/eae3084b46e2c3931db12cdef79093ad0e644bce/datasette/app.py#L1251-L1263

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Re-display user's query with an error message if an error occurs 904537568  
850057694 https://github.com/simonw/datasette/issues/619#issuecomment-850057694 https://api.github.com/repos/simonw/datasette/issues/619 MDEyOklzc3VlQ29tbWVudDg1MDA1NzY5NA== simonw 9599 2021-05-28T02:03:05Z 2021-05-28T02:03:05Z OWNER

I nearly got this working, but I ran into one last problem: the code path for when an error is raised but the user specified ?_shape=array. I'll open a draft PR with where I've got to so far.

{
    "total_count": 1,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 1,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
"Invalid SQL" page should let you edit the SQL 520655983  
850025106 https://github.com/simonw/datasette/issues/619#issuecomment-850025106 https://api.github.com/repos/simonw/datasette/issues/619 MDEyOklzc3VlQ29tbWVudDg1MDAyNTEwNg== simonw 9599 2021-05-28T00:18:30Z 2021-05-28T00:18:30Z OWNER

I'm going to return a 400 HTTP status code for "bad request", under the assumption that the client sent bad SQL.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
"Invalid SQL" page should let you edit the SQL 520655983  
850019486 https://github.com/simonw/datasette/issues/619#issuecomment-850019486 https://api.github.com/repos/simonw/datasette/issues/619 MDEyOklzc3VlQ29tbWVudDg1MDAxOTQ4Ng== simonw 9599 2021-05-28T00:05:50Z 2021-05-28T00:05:50Z OWNER
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
"Invalid SQL" page should let you edit the SQL 520655983  
849820019 https://github.com/simonw/datasette/issues/1344#issuecomment-849820019 https://api.github.com/repos/simonw/datasette/issues/1344 MDEyOklzc3VlQ29tbWVudDg0OTgyMDAxOQ== simonw 9599 2021-05-27T17:44:39Z 2021-05-27T17:52:58Z OWNER

This pattern appears to work, executed at the root of a checkout of datasette:

docker run -it -v `pwd`:/mnt --platform linux/amd64 \
  datasetteproject/datasette:0.57a1 bash -c '
    pip install "pytest>=5.2.2,<6.3.0" \
      "pytest-xdist>=2.2.1,<2.3" \
      "pytest-asyncio>=0.10,<0.16" \
      "beautifulsoup4>=4.8.1,<4.10.0" \
      "black==21.5b1" \
      "pytest-timeout>=1.4.2,<1.5" \
      "trustme>=0.7,<0.8" \
    && cd /mnt && pytest'
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Test Datasette Docker images built for different architectures 903986178  
849815361 https://github.com/simonw/datasette/issues/1344#issuecomment-849815361 https://api.github.com/repos/simonw/datasette/issues/1344 MDEyOklzc3VlQ29tbWVudDg0OTgxNTM2MQ== simonw 9599 2021-05-27T17:36:51Z 2021-05-27T17:36:51Z OWNER

To run the test suite... need to install a few more dependencies: https://github.com/simonw/datasette/blob/7b106e106000713bbee31b34d694b3dadbd4818c/setup.py#L71-L79

And then mount the tests/ folder into the container and run pytest.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Test Datasette Docker images built for different architectures 903986178  
849806581 https://github.com/simonw/datasette/issues/1344#issuecomment-849806581 https://api.github.com/repos/simonw/datasette/issues/1344 MDEyOklzc3VlQ29tbWVudDg0OTgwNjU4MQ== simonw 9599 2021-05-27T17:22:37Z 2021-05-27T17:34:27Z OWNER
~ % docker run -it --platform linux/arm/v7 -v `pwd`:/mnt -p 8001:8001 datasetteproject/datasette:0.57a1 python -c 'import platform; print(platform.uname())'
Unable to find image 'datasetteproject/datasette:0.57a1' locally
0.57a1: Pulling from datasetteproject/datasette
ee0a2cc24f29: Pull complete 
ca231153300d: Pull complete 
6073fc53d406: Pull complete 
bb2a6a6421dd: Pull complete 
14c15f441034: Pull complete 
28d113f72ba5: Pull complete 
40516c2c3785: Pull complete 
Digest: sha256:8dd469efa0b34b9e946c4e91496fc7bec702efec9c0f5ec548ccd4030f3263b2
Status: Downloaded newer image for datasetteproject/datasette:0.57a1
uname_result(system='Linux', node='c2e1a05732ed', release='4.19.121-linuxkit', version='#1 SMP Thu Jan 21 15:36:34 UTC 2021', machine='armv7l')

Shorter version:

~ % docker run -it --platform linux/ppc64le datasetteproject/datasette:0.57a1 python -c 'import platform; print(platform.uname())'
Unable to find image 'datasetteproject/datasette:0.57a1' locally
0.57a1: Pulling from datasetteproject/datasette
c840eb5e9aed: Pull complete 
fc17d8b577ec: Pull complete 
712a0ad995f8: Pull complete 
56101149914c: Pull complete 
e6241dfe9d38: Pull complete 
41ed85039888: Pull complete 
26beeabe9213: Pull complete 
Digest: sha256:8dd469efa0b34b9e946c4e91496fc7bec702efec9c0f5ec548ccd4030f3263b2
Status: Downloaded newer image for datasetteproject/datasette:0.57a1
uname_result(system='Linux', node='0fe85887fa30', release='4.19.121-linuxkit', version='#1 SMP Thu Jan 21 15:36:34 UTC 2021', machine='ppc64le')
~ % 

OK! Now I need to figure out how to run the full test suite on each one, then I can call this done.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Test Datasette Docker images built for different architectures 903986178  
849805778 https://github.com/simonw/datasette/issues/1344#issuecomment-849805778 https://api.github.com/repos/simonw/datasette/issues/1344 MDEyOklzc3VlQ29tbWVudDg0OTgwNTc3OA== simonw 9599 2021-05-27T17:21:11Z 2021-05-27T17:21:22Z OWNER

This helps:

~ % docker run -it --platform linux/s390x -v `pwd`:/mnt \
  -p 8001:8001 datasetteproject/datasette:0.57a1 \
  python -c 'import platform; print(platform.uname())'

uname_result(system='Linux', node='d14916ca91df', release='4.19.121-linuxkit', version='#1 SMP Thu Jan 21 15:36:34 UTC 2021', machine='s390x')

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Test Datasette Docker images built for different architectures 903986178  
849804684 https://github.com/simonw/datasette/issues/1344#issuecomment-849804684 https://api.github.com/repos/simonw/datasette/issues/1344 MDEyOklzc3VlQ29tbWVudDg0OTgwNDY4NA== simonw 9599 2021-05-27T17:19:16Z 2021-05-27T17:19:16Z OWNER

Having pushed that to Docker Hub the following seems to work on my laptop:

docker run --platform linux/s390x \
  -v `pwd`:/mnt -p 8001:8001 \
  datasetteproject/datasette:0.57a1 datasette -p 8001 -h 0.0.0.0

I'd like to confirm that this is correctly emulating the architecture and running the correct published image though.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Test Datasette Docker images built for different architectures 903986178  
849801969 https://github.com/simonw/datasette/issues/1344#issuecomment-849801969 https://api.github.com/repos/simonw/datasette/issues/1344 MDEyOklzc3VlQ29tbWVudDg0OTgwMTk2OQ== simonw 9599 2021-05-27T17:14:49Z 2021-05-27T17:14:49Z OWNER
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Test Datasette Docker images built for different architectures 903986178  
849797767 https://github.com/simonw/datasette/issues/1344#issuecomment-849797767 https://api.github.com/repos/simonw/datasette/issues/1344 MDEyOklzc3VlQ29tbWVudDg0OTc5Nzc2Nw== simonw 9599 2021-05-27T17:08:13Z 2021-05-27T17:08:13Z OWNER

Related issue: #1272 - being able to run the test suite inside Docker inside GitHub Actions would be incredibly helpful here.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Test Datasette Docker images built for different architectures 903986178  
849793312 https://github.com/simonw/datasette/issues/1344#issuecomment-849793312 https://api.github.com/repos/simonw/datasette/issues/1344 MDEyOklzc3VlQ29tbWVudDg0OTc5MzMxMg== simonw 9599 2021-05-27T17:01:16Z 2021-05-27T17:01:16Z OWNER
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Test Datasette Docker images built for different architectures 903986178  
849788412 https://github.com/simonw/datasette/issues/1344#issuecomment-849788412 https://api.github.com/repos/simonw/datasette/issues/1344 MDEyOklzc3VlQ29tbWVudDg0OTc4ODQxMg== simonw 9599 2021-05-27T16:53:28Z 2021-05-27T16:53:28Z OWNER

(Should also update https://docs.datasette.io/en/stable/contributing.html#release-process with notes on how this works)

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Test Datasette Docker images built for different architectures 903986178  
849788159 https://github.com/simonw/datasette/issues/1344#issuecomment-849788159 https://api.github.com/repos/simonw/datasette/issues/1344 MDEyOklzc3VlQ29tbWVudDg0OTc4ODE1OQ== simonw 9599 2021-05-27T16:53:04Z 2021-05-27T16:53:04Z OWNER

I'm going to release a 0.57a1 alpha to Docker Hub using the new buildx mechanism.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Test Datasette Docker images built for different architectures 903986178  
849785098 https://github.com/simonw/datasette/pull/1319#issuecomment-849785098 https://api.github.com/repos/simonw/datasette/issues/1319 MDEyOklzc3VlQ29tbWVudDg0OTc4NTA5OA== simonw 9599 2021-05-27T16:48:21Z 2021-05-27T16:48:21Z OWNER

OK, since I have a mechanism for deploying alpha versions now I'm going to merge this and use it to publish 0.57a1.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Add Docker multi-arch support with Buildx 881219362  
849784005 https://github.com/simonw/datasette/issues/1343#issuecomment-849784005 https://api.github.com/repos/simonw/datasette/issues/1343 MDEyOklzc3VlQ29tbWVudDg0OTc4NDAwNQ== simonw 9599 2021-05-27T16:46:37Z 2021-05-27T16:46:37Z OWNER

That worked:

 % docker run -it datasetteproject/datasette:0.57a0
Unable to find image 'datasetteproject/datasette:0.57a0' locally
0.57a0: Pulling from datasetteproject/datasette
75646c2fb410: Pull complete 
edf8b70d3cc4: Pull complete 
229daa3ebd94: Pull complete 
65d21ac50a56: Pull complete 
e083ad7cc3ca: Pull complete 
ed4de1431432: Pull complete 
1da5e8b27cca: Pull complete 
Digest: sha256:4ab61095c1d879a91d154ed469227387356a305035da9c74542c55bd500c89db
Status: Downloaded newer image for datasetteproject/datasette:0.57a0
INFO:     Started server process [1]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://127.0.0.1:8001 (Press CTRL+C to quit)
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Figure out how to publish alpha/beta releases to Docker Hub 903978133  
849783398 https://github.com/simonw/datasette/issues/1343#issuecomment-849783398 https://api.github.com/repos/simonw/datasette/issues/1343 MDEyOklzc3VlQ29tbWVudDg0OTc4MzM5OA== simonw 9599 2021-05-27T16:45:41Z 2021-05-27T16:45:41Z OWNER
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Figure out how to publish alpha/beta releases to Docker Hub 903978133  
849782098 https://github.com/simonw/datasette/issues/1343#issuecomment-849782098 https://api.github.com/repos/simonw/datasette/issues/1343 MDEyOklzc3VlQ29tbWVudDg0OTc4MjA5OA== simonw 9599 2021-05-27T16:43:40Z 2021-05-27T16:43:55Z OWNER
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Figure out how to publish alpha/beta releases to Docker Hub 903978133  
849781747 https://github.com/simonw/datasette/issues/1343#issuecomment-849781747 https://api.github.com/repos/simonw/datasette/issues/1343 MDEyOklzc3VlQ29tbWVudDg0OTc4MTc0Nw== simonw 9599 2021-05-27T16:43:04Z 2021-05-27T16:43:04Z OWNER
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Figure out how to publish alpha/beta releases to Docker Hub 903978133  
849780481 https://github.com/simonw/datasette/pull/1319#issuecomment-849780481 https://api.github.com/repos/simonw/datasette/issues/1319 MDEyOklzc3VlQ29tbWVudDg0OTc4MDQ4MQ== simonw 9599 2021-05-27T16:41:03Z 2021-05-27T16:41:03Z OWNER

It looks like all I need to do to ship an alpha version to Docker Hub is NOT point the latest tag at it after it goes live: https://github.com/simonw/datasette/blob/1a8972f9c012cd22b088c6b70661a9c3d3847853/.github/workflows/publish.yml#L75-L77

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Add Docker multi-arch support with Buildx 881219362  
849778373 https://github.com/simonw/datasette/pull/1319#issuecomment-849778373 https://api.github.com/repos/simonw/datasette/issues/1319 MDEyOklzc3VlQ29tbWVudDg0OTc3ODM3Mw== simonw 9599 2021-05-27T16:37:50Z 2021-05-27T16:37:50Z OWNER

Started a conversation about this on Twitter. Ideally I'd like to push an alpha release to Docker Hub so people can start testing these out before they make it into an official non-alpha release.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Add Docker multi-arch support with Buildx 881219362  
849762735 https://github.com/simonw/datasette/issues/1337#issuecomment-849762735 https://api.github.com/repos/simonw/datasette/issues/1337 MDEyOklzc3VlQ29tbWVudDg0OTc2MjczNQ== simonw 9599 2021-05-27T16:14:14Z 2021-05-27T16:14:14Z OWNER

Here's a demo showing that once you get above 1,000 results you still get the "..." but it no longer links to anything: https://fivethirtyeight.datasettes.com/fivethirtyeight/births%2FUS_births_2000-2014_SSA?_facet=births&_facet_size=max

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
"More" link for facets that shows _facet_size=max results 898904402  
849761894 https://github.com/simonw/datasette/issues/1332#issuecomment-849761894 https://api.github.com/repos/simonw/datasette/issues/1332 MDEyOklzc3VlQ29tbWVudDg0OTc2MTg5NA== simonw 9599 2021-05-27T16:13:07Z 2021-05-27T16:13:07Z OWNER
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
?_facet_size=X to increase number of facets results on the page 893890496  
849761508 https://github.com/simonw/datasette/issues/1337#issuecomment-849761508 https://api.github.com/repos/simonw/datasette/issues/1337 MDEyOklzc3VlQ29tbWVudDg0OTc2MTUwOA== simonw 9599 2021-05-27T16:12:31Z 2021-05-27T16:12:31Z OWNER
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
"More" link for facets that shows _facet_size=max results 898904402  
849759440 https://github.com/simonw/datasette/issues/1337#issuecomment-849759440 https://api.github.com/repos/simonw/datasette/issues/1337 MDEyOklzc3VlQ29tbWVudDg0OTc1OTQ0MA== simonw 9599 2021-05-27T16:09:34Z 2021-05-27T16:09:34Z OWNER
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
"More" link for facets that shows _facet_size=max results 898904402  
849725291 https://github.com/simonw/datasette/issues/1337#issuecomment-849725291 https://api.github.com/repos/simonw/datasette/issues/1337 MDEyOklzc3VlQ29tbWVudDg0OTcyNTI5MQ== simonw 9599 2021-05-27T15:23:33Z 2021-05-27T15:23:33Z OWNER

Turns out path_with_replaced_args() is already exposed to the table.html template context: https://github.com/simonw/datasette/blob/f1c29fd6a184254aa68efadf096bcf21e848f921/datasette/views/table.py#L897-L900

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
"More" link for facets that shows _facet_size=max results 898904402  
849724915 https://github.com/simonw/datasette/issues/1342#issuecomment-849724915 https://api.github.com/repos/simonw/datasette/issues/1342 MDEyOklzc3VlQ29tbWVudDg0OTcyNDkxNQ== simonw 9599 2021-05-27T15:23:01Z 2021-05-27T15:23:01Z OWNER

Since they are already exposed (but undocumented) this isn't actually a blocker on #1337 like I thought it was. Should still clean this up in the future though.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Improve `path_with_replaced_args()` and friends and document them 903902495  
849724108 https://github.com/simonw/datasette/issues/1342#issuecomment-849724108 https://api.github.com/repos/simonw/datasette/issues/1342 MDEyOklzc3VlQ29tbWVudDg0OTcyNDEwOA== simonw 9599 2021-05-27T15:21:54Z 2021-05-27T15:21:54Z OWNER
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Improve `path_with_replaced_args()` and friends and document them 903902495  
849722218 https://github.com/simonw/datasette/issues/1342#issuecomment-849722218 https://api.github.com/repos/simonw/datasette/issues/1342 MDEyOklzc3VlQ29tbWVudDg0OTcyMjIxOA== simonw 9599 2021-05-27T15:19:23Z 2021-05-27T15:19:23Z OWNER
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Improve `path_with_replaced_args()` and friends and document them 903902495  
849721280 https://github.com/simonw/datasette/issues/1337#issuecomment-849721280 https://api.github.com/repos/simonw/datasette/issues/1337 MDEyOklzc3VlQ29tbWVudDg0OTcyMTI4MA== simonw 9599 2021-05-27T15:18:08Z 2021-05-27T15:18:08Z OWNER

In order to cleanly implement this I need to expose the path_with_replaced_args utility function to Datasette's template engine. This is the first time this will become an exposed (and hence should-by-documented) API and I don't like its shape much.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
"More" link for facets that shows _facet_size=max results 898904402  
849708617 https://github.com/dogsheep/google-takeout-to-sqlite/pull/5#issuecomment-849708617 https://api.github.com/repos/dogsheep/google-takeout-to-sqlite/issues/5 MDEyOklzc3VlQ29tbWVudDg0OTcwODYxNw== maxhawkins 28565 2021-05-27T15:01:42Z 2021-05-27T15:01:42Z NONE

Any updates?

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
WIP: Add Gmail takeout mbox import 813880401  
849695117 https://github.com/simonw/datasette/issues/1337#issuecomment-849695117 https://api.github.com/repos/simonw/datasette/issues/1337 MDEyOklzc3VlQ29tbWVudDg0OTY5NTExNw== simonw 9599 2021-05-27T14:44:13Z 2021-05-27T14:44:13Z OWNER

Here's a much easier way to implement this: implement ?_facet_size=max (sets it to max_returned_rows, default 1000) and then link to that from the ellipsis.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
"More" link for facets that shows _facet_size=max results 898904402  
849313380 https://github.com/simonw/datasette/pull/1319#issuecomment-849313380 https://api.github.com/repos/simonw/datasette/issues/1319 MDEyOklzc3VlQ29tbWVudDg0OTMxMzM4MA== codecov[bot] 22429695 2021-05-27T04:38:44Z 2021-05-27T04:38:44Z NONE

Codecov Report

Merging #1319 (cfca570) into main (1b69753) will increase coverage by 0.02%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##             main    #1319      +/-   ##
==========================================
+ Coverage   91.51%   91.53%   +0.02%     
==========================================
  Files          34       34              
  Lines        4255     4255              
==========================================
+ Hits         3894     3895       +1     
+ Misses        361      360       -1     
<table> <thead> <tr> <th>Impacted Files</th> <th>Coverage Δ</th> <th></th> </tr> </thead> <tbody> <tr> <td>datasette/utils/__init__.py</td> <td>94.31% <0.00%> (+0.17%)</td> <td>:arrow_up:</td> </tr> </tbody> </table>

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1b69753...cfca570. Read the comment docs.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Add Docker multi-arch support with Buildx 881219362  
849311884 https://github.com/simonw/datasette/pull/1319#issuecomment-849311884 https://api.github.com/repos/simonw/datasette/issues/1319 MDEyOklzc3VlQ29tbWVudDg0OTMxMTg4NA== simonw 9599 2021-05-27T04:33:51Z 2021-05-27T04:33:51Z OWNER

This is so cool! I wonder how I can best test it? I have a Raspberry Pi I should dig out to try this.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Add Docker multi-arch support with Buildx 881219362  
849310294 https://github.com/simonw/datasette/issues/1341#issuecomment-849310294 https://api.github.com/repos/simonw/datasette/issues/1341 MDEyOklzc3VlQ29tbWVudDg0OTMxMDI5NA== simonw 9599 2021-05-27T04:29:04Z 2021-05-27T04:29:04Z OWNER

For implementation simplicity I'm going to show the item even if all of the columns are visible because ?_col= has selected all of them - selecting it will still reset the view to having no ?_col= or ?_nocol= parameters.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
"Show all columns" cog menu item should show if ?_col= is used 903200328  
849309330 https://github.com/simonw/datasette/issues/615#issuecomment-849309330 https://api.github.com/repos/simonw/datasette/issues/615 MDEyOklzc3VlQ29tbWVudDg0OTMwOTMzMA== simonw 9599 2021-05-27T04:26:10Z 2021-05-27T04:26:10Z OWNER
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
?_col= and ?_nocol= support for toggling columns on table view 517451234  
849308907 https://github.com/simonw/datasette/issues/292#issuecomment-849308907 https://api.github.com/repos/simonw/datasette/issues/292 MDEyOklzc3VlQ29tbWVudDg0OTMwODkwNw== simonw 9599 2021-05-27T04:25:01Z 2021-05-27T04:25:01Z OWNER

Now that ?_col= and ?_nocol= are implemented I'm closing this ticket - other customizations can already be handled by defining SQL views or creating canned SQL queries.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Mechanism for customizing the SQL used to select specific columns in the table view 326800219  
847207804 https://github.com/simonw/datasette/pull/1339#issuecomment-847207804 https://api.github.com/repos/simonw/datasette/issues/1339 MDEyOklzc3VlQ29tbWVudDg0NzIwNzgwNA== codecov[bot] 22429695 2021-05-24T17:22:13Z 2021-05-27T04:14:21Z NONE

Codecov Report

Merging #1339 (3d8598b) into main (a443dba) will increase coverage by 0.01%.
The diff coverage is 91.66%.

:exclamation: Current head 3d8598b differs from pull request most recent head 387c837. Consider uploading reports for the commit 387c837 to get more accurate results

@@            Coverage Diff             @@
##             main    #1339      +/-   ##
==========================================
+ Coverage   91.55%   91.56%   +0.01%     
==========================================
  Files          34       34              
  Lines        4262     4280      +18     
==========================================
+ Hits         3902     3919      +17     
- Misses        360      361       +1     
<table> <thead> <tr> <th>Impacted Files</th> <th>Coverage Δ</th> <th></th> </tr> </thead> <tbody> <tr> <td>datasette/views/table.py</td> <td>95.83% <90.00%> (-0.05%)</td> <td>:arrow_down:</td> </tr> <tr> <td>datasette/app.py</td> <td>95.84% <100.00%> (-0.01%)</td> <td>:arrow_down:</td> </tr> <tr> <td>datasette/views/database.py</td> <td>97.19% <100.00%> (ø)</td> <td></td> </tr> </tbody> </table>

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a443dba...387c837. Read the comment docs.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
?_col=/?_nocol= to show/hide columns on the table page 899851083  
849022714 https://github.com/simonw/datasette/issues/670#issuecomment-849022714 https://api.github.com/repos/simonw/datasette/issues/670 MDEyOklzc3VlQ29tbWVudDg0OTAyMjcxNA== simonw 9599 2021-05-26T18:33:47Z 2021-05-26T18:33:58Z OWNER

Worth mentioning here: I've been doing a tun of research around running Datasette-like functionality against PostgreSQL in my https://github.com/simonw/django-sql-dashboard project - which will definitely inform the Datasette implementation.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Prototoype for Datasette on PostgreSQL 564833696  
848425056 https://github.com/simonw/datasette/issues/670#issuecomment-848425056 https://api.github.com/repos/simonw/datasette/issues/670 MDEyOklzc3VlQ29tbWVudDg0ODQyNTA1Ng== river-matt 52261150 2021-05-26T03:22:32Z 2021-05-26T03:22:32Z NONE

I've also been investigating serving postgresql databases over postgrest. I like the idea of hosting some static html + js on github, but having it backed by datasets I can update and control on the database server. I started from SQLite + datasette but would like to host larger datasets (with smaller materialized views exposed publicly). I think the postgrest model where all the authorization and ownership is defined in database role grants is really powerful. But I really miss being able to define an ad-hoc query in sql, then instantly link to a json representation of it like datasette does.

P.S.: I've been sort of following along as you pop up in hacker news here and there. It's been great! Thanks for doing this all out in the open!

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Prototoype for Datasette on PostgreSQL 564833696  
847987346 https://github.com/simonw/datasette/issues/1340#issuecomment-847987346 https://api.github.com/repos/simonw/datasette/issues/1340 MDEyOklzc3VlQ29tbWVudDg0Nzk4NzM0Ng== simonw 9599 2021-05-25T15:49:55Z 2021-05-25T15:49:55Z OWNER

Usual problem with this kind of UI: what would it do on mobile where hover isn't a thing? Plenty of other questions: if it opens a menu, what options might I put in there other than "Show more like this"?

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Research: Cell action menu (like column action but for individual cells) 901009787  
847271122 https://github.com/simonw/datasette/issues/1327#issuecomment-847271122 https://api.github.com/repos/simonw/datasette/issues/1327 MDEyOklzc3VlQ29tbWVudDg0NzI3MTEyMg== GmGniap 20846286 2021-05-24T19:10:21Z 2021-05-24T19:10:21Z NONE

wow, thanks a lot @simonw , problem is solved. I converted my current json file into utf-8 format with Python script. It's working now. I'm using with Window 10.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Support Unicode characters in metadata.json 892457208  
847240470 https://github.com/simonw/datasette/issues/1327#issuecomment-847240470 https://api.github.com/repos/simonw/datasette/issues/1327 MDEyOklzc3VlQ29tbWVudDg0NzI0MDQ3MA== simonw 9599 2021-05-24T18:21:44Z 2021-05-24T18:21:44Z OWNER

I'm having trouble replicating this - I tried the following metadata.json in my local environment and it displayed correctly:

{
    "title": "မြန်မာစာ ("
}

What operating system are you using here? Is there a chance that your metadata.json file is saved in a text encoding other than utf-8?

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Support Unicode characters in metadata.json 892457208  
847237524 https://github.com/simonw/datasette/issues/1320#issuecomment-847237524 https://api.github.com/repos/simonw/datasette/issues/1320 MDEyOklzc3VlQ29tbWVudDg0NzIzNzUyNA== simonw 9599 2021-05-24T18:15:56Z 2021-05-24T18:15:56Z OWNER
{
    "total_count": 1,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 1,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Can't use apt-get in Dockerfile when using datasetteproj/datasette as base 884952179  
847232547 https://github.com/simonw/datasette/issues/1320#issuecomment-847232547 https://api.github.com/repos/simonw/datasette/issues/1320 MDEyOklzc3VlQ29tbWVudDg0NzIzMjU0Nw== simonw 9599 2021-05-24T18:06:32Z 2021-05-24T18:06:32Z OWNER

I'm testing tweaks to the Dockerfile by editing it and running:

docker build . -t datasette-057a0 --build-arg VERSION=0.57a0

Then running this to inspect the new image:

docker run -it -p 8001:8001 datasette-057a0 bash

After changing that line to rm -rf /var/lib/dpkg/info/* I can run apt-get successfully from inside that bash prompt:

# apt-get update
# apt-get install ripgrep
# rg --version
ripgrep 12.1.1
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Can't use apt-get in Dockerfile when using datasetteproj/datasette as base 884952179  
847230866 https://github.com/simonw/datasette/issues/1320#issuecomment-847230866 https://api.github.com/repos/simonw/datasette/issues/1320 MDEyOklzc3VlQ29tbWVudDg0NzIzMDg2Ng== simonw 9599 2021-05-24T18:03:29Z 2021-05-24T18:03:29Z OWNER
root@33c269900821:/# du -h /var/lib/dpkg
6.3M    /var/lib/dpkg/info
24K /var/lib/dpkg/triggers
4.0K    /var/lib/dpkg/updates
20K /var/lib/dpkg/alternatives
4.0K    /var/lib/dpkg/parts
6.7M    /var/lib/dpkg

So /var/lib/dpkg/info is most of the space, I'll try just deleting that.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Can't use apt-get in Dockerfile when using datasetteproj/datasette as base 884952179  
847223853 https://github.com/simonw/datasette/issues/1320#issuecomment-847223853 https://api.github.com/repos/simonw/datasette/issues/1320 MDEyOklzc3VlQ29tbWVudDg0NzIyMzg1Mw== simonw 9599 2021-05-24T17:51:12Z 2021-05-24T17:51:12Z OWNER

This is a good catch, thank you.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Can't use apt-get in Dockerfile when using datasetteproj/datasette as base 884952179  
846592583 https://github.com/simonw/datasette/issues/615#issuecomment-846592583 https://api.github.com/repos/simonw/datasette/issues/615 MDEyOklzc3VlQ29tbWVudDg0NjU5MjU4Mw== simonw 9599 2021-05-23T16:50:45Z 2021-05-24T17:15:48Z OWNER

Still needed:

  • Unit tests
  • Documentation

And maybe a UI mechanism for this? I could at least add a "remove this column" item to the cog menu on columns.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
?_col= and ?_nocol= support for toggling columns on table view 517451234  
846668419 https://github.com/simonw/datasette/issues/615#issuecomment-846668419 https://api.github.com/repos/simonw/datasette/issues/615 MDEyOklzc3VlQ29tbWVudDg0NjY2ODQxOQ== simonw 9599 2021-05-24T01:14:27Z 2021-05-24T01:14:27Z OWNER

I tried allowing the removal of rowid and got this exception:

  File "/Users/simon/Dropbox/Development/datasette/datasette/views/table.py", line 831, in extra_template
    display_columns, display_rows = await self.display_columns_and_rows(
  File "/Users/simon/Dropbox/Development/datasette/datasette/views/table.py", line 163, in display_columns_and_rows
    pk_path = path_from_row_pks(row, pks, not pks, False)
  File "/Users/simon/Dropbox/Development/datasette/datasette/utils/__init__.py", line 82, in path_from_row_pks
    bits = [row["rowid"]]
IndexError: No item with that key

I'm going to disable the removal of rowid - or indeed any of the primary keys, since they are needed to construct the row permalink.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
?_col= and ?_nocol= support for toggling columns on table view 517451234  
846660103 https://github.com/simonw/datasette/issues/615#issuecomment-846660103 https://api.github.com/repos/simonw/datasette/issues/615 MDEyOklzc3VlQ29tbWVudDg0NjY2MDEwMw== simonw 9599 2021-05-24T00:47:00Z 2021-05-24T00:47:00Z OWNER

Here's a bug: removing the rowid column returns an error.

Removing the rowid column should work. We can continue to show the Link column, ensuring users can still navigate to the row page for each row.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
?_col= and ?_nocol= support for toggling columns on table view 517451234  
846659272 https://github.com/simonw/datasette/issues/615#issuecomment-846659272 https://api.github.com/repos/simonw/datasette/issues/615 MDEyOklzc3VlQ29tbWVudDg0NjY1OTI3Mg== simonw 9599 2021-05-24T00:44:47Z 2021-05-24T00:44:47Z OWNER

I can't think of any good reasons to support passing the same column twice ?col=airline&_col=airline so I'm going to de-duplicate and silently ignore the second one.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
?_col= and ?_nocol= support for toggling columns on table view 517451234  
846627322 https://github.com/simonw/datasette/issues/615#issuecomment-846627322 https://api.github.com/repos/simonw/datasette/issues/615 MDEyOklzc3VlQ29tbWVudDg0NjYyNzMyMg== simonw 9599 2021-05-23T21:31:22Z 2021-05-23T21:31:22Z OWNER

Would it be useful to allow this mechanism to alias columns, for example supporting one of the following:

  • ?_col=airline as name_of_airline
  • ?_col=airline:name_of_airline

This could be handy for renaming columns to match a specific expected JSON output.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
?_col= and ?_nocol= support for toggling columns on table view 517451234  
846627163 https://github.com/simonw/datasette/issues/615#issuecomment-846627163 https://api.github.com/repos/simonw/datasette/issues/615 MDEyOklzc3VlQ29tbWVudDg0NjYyNzE2Mw== simonw 9599 2021-05-23T21:30:01Z 2021-05-23T21:30:01Z OWNER

Interesting side-effect of this implementation is that you can both control column order and request the same column multiple times:

/fivethirtyeight/airline-safety%2Fairline-safety?_col=fatal_accidents_00_14&_col=fatalities_00_14&_col=airline&_col=airline

https://user-images.githubusercontent.com/9599/119277243-58f6e080-bbd3-11eb-98be-69787cd9192d.png">

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
?_col= and ?_nocol= support for toggling columns on table view 517451234  
846626871 https://github.com/simonw/datasette/issues/615#issuecomment-846626871 https://api.github.com/repos/simonw/datasette/issues/615 MDEyOklzc3VlQ29tbWVudDg0NjYyNjg3MQ== simonw 9599 2021-05-23T21:27:36Z 2021-05-23T21:27:36Z OWNER

A better interface for this would be a full list of columns each with a checkbox for making it visible on invisible - this could then be used to apply a bulk change (rather than refreshing the interface after every removed column) and it could also be easily designed to work on narrow mobile screens where the cog icon is not visible.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
?_col= and ?_nocol= support for toggling columns on table view 517451234  
846626567 https://github.com/simonw/datasette/issues/615#issuecomment-846626567 https://api.github.com/repos/simonw/datasette/issues/615 MDEyOklzc3VlQ29tbWVudDg0NjYyNjU2Nw== simonw 9599 2021-05-23T21:25:19Z 2021-05-23T21:25:19Z OWNER

Here's a bug: removing the rowid column returns an error.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
?_col= and ?_nocol= support for toggling columns on table view 517451234  
846599732 https://github.com/simonw/datasette/issues/615#issuecomment-846599732 https://api.github.com/repos/simonw/datasette/issues/615 MDEyOklzc3VlQ29tbWVudDg0NjU5OTczMg== simonw 9599 2021-05-23T17:46:45Z 2021-05-23T17:46:45Z OWNER

I've changed my mind about forbidding ?_col= and ?_nocol= from being applied at the same time - I'm going to come up with a simple resolution rule instead.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
?_col= and ?_nocol= support for toggling columns on table view 517451234  
846595473 https://github.com/simonw/datasette/issues/615#issuecomment-846595473 https://api.github.com/repos/simonw/datasette/issues/615 MDEyOklzc3VlQ29tbWVudDg0NjU5NTQ3Mw== simonw 9599 2021-05-23T17:13:09Z 2021-05-23T17:13:09Z OWNER

Natalie suggests a quick way to implement "undo" would be to add a "Show all columns" item to that menu which only appears when at least one column is hidden.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
?_col= and ?_nocol= support for toggling columns on table view 517451234  
846595091 https://github.com/simonw/datasette/issues/615#issuecomment-846595091 https://api.github.com/repos/simonw/datasette/issues/615 MDEyOklzc3VlQ29tbWVudDg0NjU5NTA5MQ== simonw 9599 2021-05-23T17:09:59Z 2021-05-23T17:09:59Z OWNER

Here's that prototype of a "Hide this column" cog menu button:

Need a way to undo that once you've hidden a column - maybe a list of currently hidden columns that lets you un-hide them.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
?_col= and ?_nocol= support for toggling columns on table view 517451234  
846592392 https://github.com/simonw/datasette/issues/615#issuecomment-846592392 https://api.github.com/repos/simonw/datasette/issues/615 MDEyOklzc3VlQ29tbWVudDg0NjU5MjM5Mg== simonw 9599 2021-05-23T16:49:30Z 2021-05-23T16:49:44Z OWNER

I started looking at this again, inspired by #1326. I have a new diff that works against the latest main branch.

diff --git a/datasette/views/table.py b/datasette/views/table.py
index 4879228..f4b2ee2 100644
--- a/datasette/views/table.py
+++ b/datasette/views/table.py
@@ -64,6 +64,36 @@ class Row:


 class RowTableShared(DataView):
+    async def columns_to_select(self, db, table, request):
+        table_columns = await db.table_columns(table)
+        if "_col" in request.args and "_nocol" in request.args:
+            raise DatasetteError("Cannot use _col and _nocol at the same time")
+        if "_col" in request.args:
+            new_columns = []
+            for column in request.args.getlist("_col"):
+                if column not in table_columns:
+                    raise DatasetteError("_col={} is an invalid column".format(column))
+                new_columns.append(column)
+            return new_columns
+        elif "_nocol" in request.args:
+            # Return all columns EXCEPT these
+            bad_columns = [
+                column
+                for column in request.args.getlist("_nocol")
+                if column not in table_columns
+            ]
+            if bad_columns:
+                raise DatasetteError(
+                    "_nocol={} - invalid columns".format(", ".join(bad_columns))
+                )
+            return [
+                column
+                for column in table_columns
+                if column not in request.args.getlist("_nocol")
+            ]
+        else:
+            return table_columns
+
     async def sortable_columns_for_table(self, database, table, use_rowid):
         db = self.ds.databases[database]
         table_metadata = self.ds.table_metadata(database, table)
@@ -321,18 +351,16 @@ class TableView(RowTableShared):
         )

         pks = await db.primary_keys(table)
-        table_column_details = await db.table_column_details(table)
-        table_columns = [column.name for column in table_column_details]
-
-        select_columns = ", ".join(escape_sqlite(t) for t in table_columns)
+        table_columns = await self.columns_to_select(db, table, request)
+        select_clause = ", ".join(escape_sqlite(t) for t in table_columns)

         use_rowid = not pks and not is_view
         if use_rowid:
-            select = f"rowid, {select_columns}"
+            select = f"rowid, {select_clause}"
             order_by = "rowid"
             order_by_pks = "rowid"
         else:
-            select = select_columns
+            select = select_clause
             order_by_pks = ", ".join([escape_sqlite(pk) for pk in pks])
             order_by = order_by_pks

@@ -715,6 +743,8 @@ class TableView(RowTableShared):
                 column = fk["column"]
                 if column not in columns_to_expand:
                     continue
+                if column not in columns:
+                    continue
                 expanded_columns.append(column)
                 # Gather the values
                 column_index = columns.index(column)
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
?_col= and ?_nocol= support for toggling columns on table view 517451234  
846493370 https://github.com/simonw/datasette/issues/1326#issuecomment-846493370 https://api.github.com/repos/simonw/datasette/issues/1326 MDEyOklzc3VlQ29tbWVudDg0NjQ5MzM3MA== simonw 9599 2021-05-23T02:55:00Z 2021-05-23T02:55:00Z OWNER

Currently you can do this with a custom SQL query, but as you've noticed this means you don't get automatic pagination: https://latest.datasette.io/fixtures.json?sql=select+pk%2C+created%2C+state+from+facetable+order+by+pk+limit+10&_shape=array

I have an open issue to allow selecting specific columns on the table page: #615

A workaround you can use today is to define a SQL view that selects just the tables you want. View pages behave exactly like tables in that they provide pagination by default.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
How to limit fields returned from the JSON API? 891969037  
846483426 https://github.com/simonw/datasette/issues/1331#issuecomment-846483426 https://api.github.com/repos/simonw/datasette/issues/1331 MDEyOklzc3VlQ29tbWVudDg0NjQ4MzQyNg== simonw 9599 2021-05-23T00:57:30Z 2021-05-23T00:57:30Z OWNER

Here's an alpha release with updated dependencies: https://pypi.org/project/datasette/0.57a0/

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Add support for Jinja2 version 3.0 893537744  
846482057 https://github.com/simonw/datasette/issues/1331#issuecomment-846482057 https://api.github.com/repos/simonw/datasette/issues/1331 MDEyOklzc3VlQ29tbWVudDg0NjQ4MjA1Nw== simonw 9599 2021-05-23T00:39:55Z 2021-05-23T00:39:55Z OWNER

I'm stuck also because datasette wants itsdangerous~=1.1 instead of allowing itsdangerous-2.0.0

Bumped that dependency in b64d87204612a84663616e075f542499a5d82a03

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Add support for Jinja2 version 3.0 893537744  
846481937 https://github.com/simonw/datasette/issues/1331#issuecomment-846481937 https://api.github.com/repos/simonw/datasette/issues/1331 MDEyOklzc3VlQ29tbWVudDg0NjQ4MTkzNw== simonw 9599 2021-05-23T00:38:28Z 2021-05-23T00:38:28Z OWNER

I'm going to push an alpha release with the new dependencies shortly.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Add support for Jinja2 version 3.0 893537744  
846481896 https://github.com/simonw/datasette/issues/1337#issuecomment-846481896 https://api.github.com/repos/simonw/datasette/issues/1337 MDEyOklzc3VlQ29tbWVudDg0NjQ4MTg5Ng== simonw 9599 2021-05-23T00:37:52Z 2021-05-23T00:37:52Z OWNER

The tricky thing about this is that ideally I'd like this to link to a paginated set of results - but arbitrary SQL queries do not yet support pagination, see #856.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
"More" link for facets that shows _facet_size=max results 898904402  
846481801 https://github.com/simonw/datasette/issues/1332#issuecomment-846481801 https://api.github.com/repos/simonw/datasette/issues/1332 MDEyOklzc3VlQ29tbWVudDg0NjQ4MTgwMQ== simonw 9599 2021-05-23T00:36:20Z 2021-05-23T00:36:20Z OWNER
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
?_facet_size=X to increase number of facets results on the page 893890496  
846479730 https://github.com/simonw/datasette/issues/1332#issuecomment-846479730 https://api.github.com/repos/simonw/datasette/issues/1332 MDEyOklzc3VlQ29tbWVudDg0NjQ3OTczMA== simonw 9599 2021-05-23T00:11:31Z 2021-05-23T00:11:31Z OWNER

Here's the code in question: https://github.com/simonw/datasette/blob/5e9672c9bb33e41686472db4aa427168f9e67dbe/datasette/facets.py#L187-L210

It has access to the current request object as self.request.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
?_facet_size=X to increase number of facets results on the page 893890496  
846479212 https://github.com/simonw/datasette/issues/1332#issuecomment-846479212 https://api.github.com/repos/simonw/datasette/issues/1332 MDEyOklzc3VlQ29tbWVudDg0NjQ3OTIxMg== simonw 9599 2021-05-23T00:07:46Z 2021-05-23T00:07:46Z OWNER

?_facet_size=50 seems like a very reasonable thing to support - I'll cap it at whatever the max_returned_rows setting is (which defaults to 1000).

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
?_facet_size=X to increase number of facets results on the page 893890496  
846479062 https://github.com/simonw/datasette/issues/1332#issuecomment-846479062 https://api.github.com/repos/simonw/datasette/issues/1332 MDEyOklzc3VlQ29tbWVudDg0NjQ3OTA2Mg== simonw 9599 2021-05-23T00:06:34Z 2021-05-23T00:06:34Z OWNER

A URL parameter to modify that facet size is a really good idea.

I thought I had an issue open for "..." linking to more results but I can't find it now.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
?_facet_size=X to increase number of facets results on the page 893890496  
846413174 https://github.com/dogsheep/github-to-sqlite/pull/59#issuecomment-846413174 https://api.github.com/repos/dogsheep/github-to-sqlite/issues/59 MDEyOklzc3VlQ29tbWVudDg0NjQxMzE3NA== frosencrantz 631242 2021-05-22T14:06:19Z 2021-05-22T14:06:19Z CONTRIBUTOR

Thanks Simon!

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Remove unneeded exists=True for -a/--auth flag. 771872303  
846138580 https://github.com/simonw/datasette/pull/1325#issuecomment-846138580 https://api.github.com/repos/simonw/datasette/issues/1325 MDEyOklzc3VlQ29tbWVudDg0NjEzODU4MA== stonebig 4312421 2021-05-21T18:00:10Z 2021-05-21T18:00:10Z NONE

would be nice to have

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Update itsdangerous requirement from ~=1.1 to >=1.1,<3.0 890073989  
846137332 https://github.com/simonw/datasette/issues/1331#issuecomment-846137332 https://api.github.com/repos/simonw/datasette/issues/1331 MDEyOklzc3VlQ29tbWVudDg0NjEzNzMzMg== stonebig 4312421 2021-05-21T17:57:53Z 2021-05-21T17:57:53Z NONE

I'm stuck also because datasette wants itsdangerous~=1.1 instead of allowing itsdangerous-2.0.0

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Add support for Jinja2 version 3.0 893537744  
845985439 https://github.com/simonw/datasette/issues/856#issuecomment-845985439 https://api.github.com/repos/simonw/datasette/issues/856 MDEyOklzc3VlQ29tbWVudDg0NTk4NTQzOQ== bram2000 5268174 2021-05-21T14:22:41Z 2021-05-21T14:22:41Z NONE

Thanks Simon this is working very well.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Consider pagination of canned queries 642296989  
844970776 https://github.com/simonw/datasette/issues/1331#issuecomment-844970776 https://api.github.com/repos/simonw/datasette/issues/1331 MDEyOklzc3VlQ29tbWVudDg0NDk3MDc3Ng== MarkusH 475613 2021-05-20T10:40:25Z 2021-05-20T10:40:25Z NONE

Any chance you could push a new datasette release with the updated dependencies in the setup.py, @simonw?

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Add support for Jinja2 version 3.0 893537744  
844250232 https://github.com/dogsheep/github-to-sqlite/pull/59#issuecomment-844250232 https://api.github.com/repos/dogsheep/github-to-sqlite/issues/59 MDEyOklzc3VlQ29tbWVudDg0NDI1MDIzMg== simonw 9599 2021-05-19T16:08:10Z 2021-05-19T16:08:10Z MEMBER

Thanks for catching this.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Remove unneeded exists=True for -a/--auth flag. 771872303  
844249385 https://github.com/dogsheep/github-to-sqlite/pull/61#issuecomment-844249385 https://api.github.com/repos/dogsheep/github-to-sqlite/issues/61 MDEyOklzc3VlQ29tbWVudDg0NDI0OTM4NQ== simonw 9599 2021-05-19T16:07:06Z 2021-05-19T16:07:06Z MEMBER

Thanks!

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
fixing typo in get cli help text 797108702  
844007155 https://github.com/simonw/datasette/pull/1335#issuecomment-844007155 https://api.github.com/repos/simonw/datasette/issues/1335 MDEyOklzc3VlQ29tbWVudDg0NDAwNzE1NQ== codecov[bot] 22429695 2021-05-19T11:21:45Z 2021-05-19T11:21:45Z NONE

Codecov Report

Merging #1335 (d91c18d) into main (9b3b7e2) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main    #1335   +/-   ##
=======================================
  Coverage   91.53%   91.53%           
=======================================
  Files          34       34           
  Lines        4255     4255           
=======================================
  Hits         3895     3895           
  Misses        360      360           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9b3b7e2...d91c18d. Read the comment docs.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Fix small typo 895315478  

Next page

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options:

CREATE TABLE [issue_comments] (
   [html_url] TEXT,
   [issue_url] TEXT,
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [created_at] TEXT,
   [updated_at] TEXT,
   [author_association] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [issue] INTEGER REFERENCES [issues]([id])
, [performed_via_github_app] TEXT);
CREATE INDEX [idx_issue_comments_issue]
                ON [issue_comments] ([issue]);
CREATE INDEX [idx_issue_comments_user]
                ON [issue_comments] ([user]);