5,580 rows sorted by updated_at descending

View and edit SQL

Suggested facets: reactions, updated_at (date)

author_association

id html_url issue_url node_id user created_at updated_at ▲ author_association body reactions issue performed_via_github_app
837166862 https://github.com/simonw/datasette/issues/1280#issuecomment-837166862 https://api.github.com/repos/simonw/datasette/issues/1280 MDEyOklzc3VlQ29tbWVudDgzNzE2Njg2Mg== blairdrummond 10801138 2021-05-10T19:07:46Z 2021-05-10T19:07:46Z NONE

Do you have a list of sqlite versions you want to test against?

One cool thing I saw recently (that we started using) was using import docker within python, and then writing pytest functions which executed against the container

setup

example

The inspiration for this came from the jupyter docker-stacks

So off the top of my head, could look at building the container with different sqlite versions as a build-arg, then run tests against the containers. Just brainstorming though

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Ability to run CI against multiple SQLite versions 842862708  
835491318 https://github.com/simonw/datasette/pull/1296#issuecomment-835491318 https://api.github.com/repos/simonw/datasette/issues/1296 MDEyOklzc3VlQ29tbWVudDgzNTQ5MTMxOA== blairdrummond 10801138 2021-05-08T19:59:01Z 2021-05-08T19:59:01Z NONE

I have also found that ubuntu has fewer vulnerabilities than the buster based images.

➜  ~ docker pull python:3-buster
➜  ~ trivy image python:3-buster | head                             
2021-04-28T17:14:29.313-0400    INFO    Detecting Debian vulnerabilities...
2021-04-28T17:14:29.393-0400    INFO    Trivy skips scanning programming language libraries because no supported file was detected
python:3-buster (debian 10.9)
=============================
Total: 1621 (UNKNOWN: 13, LOW: 1106, MEDIUM: 343, HIGH: 145, CRITICAL: 14)
+------------------------------+---------------------+----------+------------------------------+---------------+--------------------------------------------------------------+
|           LIBRARY            |  VULNERABILITY ID   | SEVERITY |      INSTALLED VERSION       | FIXED VERSION |                            TITLE                             |
+------------------------------+---------------------+----------+------------------------------+---------------+--------------------------------------------------------------+
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Dockerfile: use Ubuntu 20.10 as base 855446829  
834636796 https://github.com/simonw/datasette/issues/1168#issuecomment-834636796 https://api.github.com/repos/simonw/datasette/issues/1168 MDEyOklzc3VlQ29tbWVudDgzNDYzNjc5Ng== simonw 9599 2021-05-07T17:22:52Z 2021-05-07T17:22:52Z OWNER

Related: Here's an implementation of a get_metadata() plugin hook by @brandonrobertz https://github.com/next-LI/datasette/commit/3fd8ce91f3108c82227bf65ff033929426c60437

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Mechanism for storing metadata in _metadata tables 777333388  
833132571 https://github.com/simonw/datasette/issues/1300#issuecomment-833132571 https://api.github.com/repos/simonw/datasette/issues/1300 MDEyOklzc3VlQ29tbWVudDgzMzEzMjU3MQ== abdusco 3243482 2021-05-06T00:16:50Z 2021-05-06T00:18:05Z CONTRIBUTOR

I ended up using some JS as a workaround.

First, add a JS file in metadata.yaml:

extra_js_urls:
  - '/static/app.js'

then inside the script, find the blob download links and replace .blob extension in the url with .jpg and replace the links with <img/> elements.
You need to add an output formatter to serve BLOB columns as JPG. You can find the code in the first post.
<del>Replacing .blob -> .jpg might not even be necessary, because browsers only care about the mime type, so you only need to serve the binary content with the right content-type header.</del>. You need to replace the extension, otherwise the output renderer will not run.

window.addEventListener('DOMContentLoaded', () => {
    function renderBlobImages() {
        document.querySelectorAll('a[href*=".blob"]').forEach(el => {
            const img = document.createElement('img');
            img.className = 'blob-image';
            img.loading = 'lazy';
            img.src = el.href.replace('.blob', '.jpg');
            el.parentElement.replaceChild(img, el);
        });
    }

    renderBlobImages();
});

while this does the job, I'd prefer handling this in Python where it belongs.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Generating URL for a row inside `render_cell` hook 860625833  
832676649 https://github.com/simonw/datasette/pull/1318#issuecomment-832676649 https://api.github.com/repos/simonw/datasette/issues/1318 MDEyOklzc3VlQ29tbWVudDgzMjY3NjY0OQ== codecov[bot] 22429695 2021-05-05T13:13:45Z 2021-05-05T13:13:45Z NONE

Codecov Report

Merging #1318 (e06c099) into main (1b69753) will increase coverage by 0.02%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##             main    #1318      +/-   ##
==========================================
+ Coverage   91.51%   91.53%   +0.02%     
==========================================
  Files          34       34              
  Lines        4255     4255              
==========================================
+ Hits         3894     3895       +1     
+ Misses        361      360       -1     
<table> <thead> <tr> <th>Impacted Files</th> <th>Coverage Δ</th> <th></th> </tr> </thead> <tbody> <tr> <td>datasette/utils/__init__.py</td> <td>94.31% <0.00%> (+0.17%)</td> <td>:arrow_up:</td> </tr> </tbody> </table>

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1b69753...e06c099. Read the comment docs.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Bump black from 21.4b2 to 21.5b0 876431852  
831004775 https://github.com/dogsheep/genome-to-sqlite/issues/1#issuecomment-831004775 https://api.github.com/repos/dogsheep/genome-to-sqlite/issues/1 MDEyOklzc3VlQ29tbWVudDgzMTAwNDc3NQ== cobiadigital 25372415 2021-05-03T03:46:23Z 2021-05-03T03:46:23Z NONE

RS1800955 is related to novelty seeking and ADHD
https://www.snpedia.com/index.php/Rs1800955

select rsid, genotype, case genotype when 'CC' then 'increased susceptibility to novelty seeking' when 'CT' then 'increased susceptibility to novelty seeking' when 'TT' then 'normal' end as interpretation from genome where rsid = 'rs1800955'

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Figure out some interesting example SQL queries 496415321  
829885904 https://github.com/simonw/datasette/issues/1310#issuecomment-829885904 https://api.github.com/repos/simonw/datasette/issues/1310 MDEyOklzc3VlQ29tbWVudDgyOTg4NTkwNA== ColinMaudry 3747136 2021-04-30T06:58:46Z 2021-04-30T07:26:11Z NONE

I made it work with openpyxl. I'm not sure all the code under @hookimpl is necessary... but it works :)

from datasette import hookimpl
from datasette.utils.asgi import Response
from openpyxl import Workbook
from openpyxl.writer.excel import save_virtual_workbook
from openpyxl.cell import WriteOnlyCell
from openpyxl.styles import Alignment, Font, PatternFill
from tempfile import NamedTemporaryFile

def render_spreadsheet(rows):
    wb = Workbook(write_only=True)
    ws = wb.create_sheet()
    ws = wb.active
    ws.title = "decp"

    columns = rows[0].keys()
    headers = []
    for col in columns :
        c = WriteOnlyCell(ws, col)
        c.fill = PatternFill("solid", fgColor="DDEFFF")
        headers.append(c)
    ws.append(headers)

    for row in rows:
        wsRow = []
        for col in columns:
            c = WriteOnlyCell(ws, row[col])
            if col == "objet" :
                c.alignment = Alignment(wrapText = True)
            wsRow.append(c)
        ws.append(wsRow)

    with NamedTemporaryFile() as tmp:
        wb.save(tmp.name)
        tmp.seek(0)
        return Response(
            tmp.read(),
            headers={
                'Content-Disposition': 'attachment; filename=decp.xlsx',
                'Content-type': 'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet'
            }
        )

@hookimpl
def register_output_renderer():
    return {"extension": "xlsx",
    "render": render_spreadsheet,
    "can_render": lambda: False}

The key part was to find the right function to wrap the spreadsheet object wb. NamedTemporaryFile() did it!

I'll update this issue when the plugin is packaged and ready for broader use.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
I'm creating a plugin to export a spreadsheet file (.ods or .xlsx) 870125126  
829352402 https://github.com/simonw/datasette/pull/1313#issuecomment-829352402 https://api.github.com/repos/simonw/datasette/issues/1313 MDEyOklzc3VlQ29tbWVudDgyOTM1MjQwMg== dependabot-preview[bot] 27856297 2021-04-29T15:47:23Z 2021-04-29T15:47:23Z CONTRIBUTOR

This pull request will no longer be automatically closed when a new version is found as this pull request was created by Dependabot Preview and this repo is using a version: 2 config file. You can close this pull request and let Dependabot re-create it the next time it checks for updates.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Bump black from 20.8b1 to 21.4b2 871046111  
829349118 https://github.com/simonw/datasette/pull/1314#issuecomment-829349118 https://api.github.com/repos/simonw/datasette/issues/1314 MDEyOklzc3VlQ29tbWVudDgyOTM0OTExOA== codecov[bot] 22429695 2021-04-29T15:43:32Z 2021-04-29T15:43:32Z NONE

Codecov Report

Merging #1314 (98eea0b) into main (a4bb2ab) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main    #1314   +/-   ##
=======================================
  Coverage   91.51%   91.51%           
=======================================
  Files          34       34           
  Lines        4255     4255           
=======================================
  Hits         3894     3894           
  Misses        361      361           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a4bb2ab...98eea0b. Read the comment docs.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Upgrade to GitHub-native Dependabot 871157602  
829265979 https://github.com/simonw/datasette/pull/1313#issuecomment-829265979 https://api.github.com/repos/simonw/datasette/issues/1313 MDEyOklzc3VlQ29tbWVudDgyOTI2NTk3OQ== codecov[bot] 22429695 2021-04-29T14:04:13Z 2021-04-29T14:04:13Z NONE

Codecov Report

Merging #1313 (3cd7ad4) into main (a4bb2ab) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main    #1313   +/-   ##
=======================================
  Coverage   91.51%   91.51%           
=======================================
  Files          34       34           
  Lines        4255     4255           
=======================================
  Hits         3894     3894           
  Misses        361      361           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a4bb2ab...3cd7ad4. Read the comment docs.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Bump black from 20.8b1 to 21.4b2 871046111  
829260725 https://github.com/simonw/datasette/pull/1311#issuecomment-829260725 https://api.github.com/repos/simonw/datasette/issues/1311 MDEyOklzc3VlQ29tbWVudDgyOTI2MDcyNQ== dependabot-preview[bot] 27856297 2021-04-29T13:58:08Z 2021-04-29T13:58:08Z CONTRIBUTOR

Superseded by #1313.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Bump black from 20.8b1 to 21.4b1 870227815  
828683322 https://github.com/simonw/datasette/pull/1311#issuecomment-828683322 https://api.github.com/repos/simonw/datasette/issues/1311 MDEyOklzc3VlQ29tbWVudDgyODY4MzMyMg== codecov[bot] 22429695 2021-04-28T18:30:49Z 2021-04-28T18:30:49Z NONE

Codecov Report

Merging #1311 (baf3030) into main (a4bb2ab) will increase coverage by 0.07%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##             main    #1311      +/-   ##
==========================================
+ Coverage   91.51%   91.58%   +0.07%     
==========================================
  Files          34       34              
  Lines        4255     4255              
==========================================
+ Hits         3894     3897       +3     
+ Misses        361      358       -3     
<table> <thead> <tr> <th>Impacted Files</th> <th>Coverage Δ</th> <th></th> </tr> </thead> <tbody> <tr> <td>datasette/database.py</td> <td>93.68% <0.00%> (+0.74%)</td> <td>:arrow_up:</td> </tr> <tr> <td>datasette/views/index.py</td> <td>98.18% <0.00%> (+1.81%)</td> <td>:arrow_up:</td> </tr> </tbody> </table>

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a4bb2ab...baf3030. Read the comment docs.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Bump black from 20.8b1 to 21.4b1 870227815  
828679943 https://github.com/simonw/datasette/pull/1309#issuecomment-828679943 https://api.github.com/repos/simonw/datasette/issues/1309 MDEyOklzc3VlQ29tbWVudDgyODY3OTk0Mw== dependabot-preview[bot] 27856297 2021-04-28T18:26:03Z 2021-04-28T18:26:03Z CONTRIBUTOR

Superseded by #1311.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Bump black from 20.8b1 to 21.4b0 869237023  
828670621 https://github.com/simonw/datasette/issues/1310#issuecomment-828670621 https://api.github.com/repos/simonw/datasette/issues/1310 MDEyOklzc3VlQ29tbWVudDgyODY3MDYyMQ== ColinMaudry 3747136 2021-04-28T18:12:08Z 2021-04-28T18:12:08Z NONE

Apparently, beside a string, Reponse could also work with bytes.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
I'm creating a plugin to export a spreadsheet file (.ods or .xlsx) 870125126  
827911909 https://github.com/simonw/datasette/pull/1309#issuecomment-827911909 https://api.github.com/repos/simonw/datasette/issues/1309 MDEyOklzc3VlQ29tbWVudDgyNzkxMTkwOQ== codecov[bot] 22429695 2021-04-27T20:35:15Z 2021-04-27T20:35:15Z NONE

Codecov Report

Merging #1309 (20fc3fe) into main (a4bb2ab) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main    #1309   +/-   ##
=======================================
  Coverage   91.51%   91.51%           
=======================================
  Files          34       34           
  Lines        4255     4255           
=======================================
  Hits         3894     3894           
  Misses        361      361           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a4bb2ab...20fc3fe. Read the comment docs.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Bump black from 20.8b1 to 21.4b0 869237023  
826784306 https://github.com/simonw/datasette/issues/173#issuecomment-826784306 https://api.github.com/repos/simonw/datasette/issues/173 MDEyOklzc3VlQ29tbWVudDgyNjc4NDMwNg== ColinMaudry 3747136 2021-04-26T12:10:01Z 2021-04-26T12:10:01Z NONE

I found a neat tutorial to set up gettext with jinja2: http://siongui.github.io/2016/01/17/i18n-python-web-application-by-gettext-jinja2/

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
I18n and L10n support 281110295  
826041458 https://github.com/simonw/datasette/issues/1308#issuecomment-826041458 https://api.github.com/repos/simonw/datasette/issues/1308 MDEyOklzc3VlQ29tbWVudDgyNjA0MTQ1OA== simonw 9599 2021-04-24T06:07:01Z 2021-04-24T06:07:01Z OWNER

I can use td.type-pk instead - here's the existing HTML:

<td class="col-Link type-pk"><a href="/link/link/1">1</a></td>
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Columns named "link" display in bold 866668415  
826040909 https://github.com/simonw/datasette/issues/1308#issuecomment-826040909 https://api.github.com/repos/simonw/datasette/issues/1308 MDEyOklzc3VlQ29tbWVudDgyNjA0MDkwOQ== simonw 9599 2021-04-24T06:01:21Z 2021-04-24T06:01:21Z OWNER
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Columns named "link" display in bold 866668415  
826040676 https://github.com/simonw/datasette/issues/1308#issuecomment-826040676 https://api.github.com/repos/simonw/datasette/issues/1308 MDEyOklzc3VlQ29tbWVudDgyNjA0MDY3Ng== simonw 9599 2021-04-24T05:58:35Z 2021-04-24T05:58:35Z OWNER
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Columns named "link" display in bold 866668415  
824866566 https://github.com/simonw/datasette/pull/1306#issuecomment-824866566 https://api.github.com/repos/simonw/datasette/issues/1306 MDEyOklzc3VlQ29tbWVudDgyNDg2NjU2Ng== codecov[bot] 22429695 2021-04-22T13:59:04Z 2021-04-22T13:59:04Z NONE

Codecov Report

Merging #1306 (115332c) into main (6ed9238) will increase coverage by 0.00%.
The diff coverage is 100.00%.

@@           Coverage Diff           @@
##             main    #1306   +/-   ##
=======================================
  Coverage   91.51%   91.51%           
=======================================
  Files          34       34           
  Lines        4255     4256    +1     
=======================================
+ Hits         3894     3895    +1     
  Misses        361      361           
<table> <thead> <tr> <th>Impacted Files</th> <th>Coverage Δ</th> <th></th> </tr> </thead> <tbody> <tr> <td>datasette/views/index.py</td> <td>96.42% <100.00%> (+0.06%)</td> <td>:arrow_up:</td> </tr> </tbody> </table>

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6ed9238...115332c. Read the comment docs.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Possible fix for issue #1305 864979486  
823961091 https://github.com/simonw/datasette/issues/173#issuecomment-823961091 https://api.github.com/repos/simonw/datasette/issues/173 MDEyOklzc3VlQ29tbWVudDgyMzk2MTA5MQ== ColinMaudry 3747136 2021-04-21T10:37:05Z 2021-04-21T10:37:36Z NONE

I have the feeling that the text visible to users is 95% present in template files (datasette/templates). The python code mainly contains error messages.

In the current situation, the best way to provide a localized frontend is to translate the templates and configure datasette to use them. I think I'm going to do it for French.

If we want localization to be better integrated, for the python code, I think gettext is the way to go. The .po can be translated in user-friendly tools such as Transifex and Crowdin.

For the templates, I'm not sure how we could do it cleanly and easy to maintain. Maybe the tools above could parse HTML and detect the strings to be translated.

In any case, localization implementing l10n is just the first step: a continuous process must be setup to maintain the translations and produce new ones while datasette keeps getting new features.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
I18n and L10n support 281110295  
823102978 https://github.com/simonw/datasette/issues/1298#issuecomment-823102978 https://api.github.com/repos/simonw/datasette/issues/1298 MDEyOklzc3VlQ29tbWVudDgyMzEwMjk3OA== dracos 154364 2021-04-20T08:51:23Z 2021-04-20T08:51:23Z NONE
  1. Max height would still let you scroll the page to underneath the facets to the table, but would mean the table would never take up more than your window size, so the horizontal scrollbar would be visible as soon as the table took up the size of the window.
  2. Yes, this wouldn't be for mobile :) It'd be desktop-only styling. On mobile you can scroll much more easily with touch, anyway. In your case, perhaps better would be the whole top half would be facets, bottom left quadrant chart, bottom right table. Depends upon the particular use case, as you say.
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
improve table horizontal scroll experience 855476501  
823093669 https://github.com/simonw/datasette/issues/1298#issuecomment-823093669 https://api.github.com/repos/simonw/datasette/issues/1298 MDEyOklzc3VlQ29tbWVudDgyMzA5MzY2OQ== mroswell 192568 2021-04-20T08:38:10Z 2021-04-20T08:40:22Z CONTRIBUTOR

@dracos I appreciate your ideas!

  1. Ooh, I like this: https://codepen.io/astro87/pen/LYRQNbd?editors=1100 (That's the codepen from your linked stackoverflow.)
  2. I worry that a max height will be a problem when my facets are open. (I've got 35 active ingredients, and so I've set the default_facet_size to 35.)
  3. I don't understand this one. I'm observing the screenshot... very helpful! (Ah, okay, TR = Top Right and BR = Bottom Right. Absolute grid refers to position style.) All the scroll bars look a little wonky to me. I've also got a lot of facets, and prefer the extra horizontal space so that not as many facets disappear below the fold. My site also has end users... some will be on mobile... not sure what the absolute grid would do there...
  4. (I still think a hover-arrow that scrolls upon click would help, too...)

But meanwhile, I'm going to go ahead and see if I can apply that shadow. (Never would've thought of that.) Hmmm... I'm not an SCSS person. This looks helpful! https://jsonformatter.org/scss-to-css

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
improve table horizontal scroll experience 855476501  
823064725 https://github.com/simonw/datasette/issues/1298#issuecomment-823064725 https://api.github.com/repos/simonw/datasette/issues/1298 MDEyOklzc3VlQ29tbWVudDgyMzA2NDcyNQ== dracos 154364 2021-04-20T07:57:14Z 2021-04-20T07:57:14Z NONE

My suggestions, originally made on twitter, but might be better here now:

  1. Could have a CSS shadow (one of the comments on https://stackoverflow.com/questions/44793453/how-do-i-add-a-top-and-bottom-shadow-while-scrolling-but-only-when-needed is a codepen for horizontal instead of vertical);

  2. Could give the table a max-height (either the window or work out the available space) so that it is both vertically/horizontally scrollable and you don't have to scroll to the bottom in order to see this;

  3. On a desktop browser, what I think I'd want is an absolute grid to work with - left query/filters, TR chart (or map), BR table. No problem with scrolling then. Here is a mockup I made when this was about the map plugin:

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
improve table horizontal scroll experience 855476501  
822486113 https://github.com/simonw/datasette/pull/1303#issuecomment-822486113 https://api.github.com/repos/simonw/datasette/issues/1303 MDEyOklzc3VlQ29tbWVudDgyMjQ4NjExMw== codecov[bot] 22429695 2021-04-19T13:55:24Z 2021-04-19T13:55:24Z NONE

Codecov Report

Merging #1303 (c348ff1) into main (0a7621f) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main    #1303   +/-   ##
=======================================
  Coverage   91.51%   91.51%           
=======================================
  Files          34       34           
  Lines        4255     4255           
=======================================
  Hits         3894     3894           
  Misses        361      361           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0a7621f...c348ff1. Read the comment docs.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Update pytest-asyncio requirement from <0.15,>=0.10 to >=0.10,<0.16 861331159  
821971059 https://github.com/simonw/datasette/issues/1300#issuecomment-821971059 https://api.github.com/repos/simonw/datasette/issues/1300 MDEyOklzc3VlQ29tbWVudDgyMTk3MTA1OQ== abdusco 3243482 2021-04-18T10:42:19Z 2021-04-18T10:42:19Z CONTRIBUTOR

If there's a simpler way to generate a URL for a specific row, I'm all ears

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Generating URL for a row inside `render_cell` hook 860625833  
821970965 https://github.com/simonw/datasette/issues/1300#issuecomment-821970965 https://api.github.com/repos/simonw/datasette/issues/1300 MDEyOklzc3VlQ29tbWVudDgyMTk3MDk2NQ== abdusco 3243482 2021-04-18T10:41:15Z 2021-04-18T10:41:15Z CONTRIBUTOR

If I change the hookspec and add a row parameter, it works

https://github.com/simonw/datasette/blob/7a2ed9f8a119e220b66d67c7b9e07cbab47b1196/datasette/hookspecs.py#L58

def render_cell(value, column, row, table, database, datasette):

But to generate a URL, I need the primary keys, but I can't call pks = await db.primary_keys(table) inside a sync function. I can't call datasette.utils.detect_primary_keys either, because the db connection is not publicly exposed (AFAICT).

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Generating URL for a row inside `render_cell` hook 860625833  
819775388 https://github.com/simonw/datasette/issues/1196#issuecomment-819775388 https://api.github.com/repos/simonw/datasette/issues/1196 MDEyOklzc3VlQ29tbWVudDgxOTc3NTM4OA== robroc 1219001 2021-04-14T19:28:38Z 2021-04-14T19:28:38Z NONE

@QAInsights I'm having a similar problem when publishing to Cloud Run on Windows. It's not able to access certain packages in my conda environment where Datasette is installed. Can you explain how you got it to work in WSL? Were you able to access the .db file in the Windows file system? Thank you.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Access Denied Error in Windows 791237799  
819467759 https://github.com/simonw/datasette/pull/1296#issuecomment-819467759 https://api.github.com/repos/simonw/datasette/issues/1296 MDEyOklzc3VlQ29tbWVudDgxOTQ2Nzc1OQ== camallen 295329 2021-04-14T12:07:37Z 2021-04-14T12:11:36Z CONTRIBUTOR

Removing /var/lib/apt and /var/lib/dpkg makes apt and dpkg unusable in
images based on this one. Running apt-get clean and removing
/var/lib/apt/lists achieves similar size savings.

this PR helps me as removing the /var/lib/apt and /var/lib/dpkg directories breaks my ability to add packages when using datasetteproject/datasette:0.56 as a base image.


Shorterm workaround for me was to use this in my Dockerfile

FROM datasetteproject/datasette:0.56

RUN mkdir -p /var/lib/apt
RUN mkdir -p /var/lib/dpkg
RUN mkdir -p /var/lib/dpkg/updates
RUN mkdir -p /var/lib/dpkg/info
RUN touch /var/lib/dpkg/status

RUN apt-get update # and install your packages etc
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Dockerfile: use Ubuntu 20.10 as base 855446829  
817414881 https://github.com/simonw/datasette/issues/830#issuecomment-817414881 https://api.github.com/repos/simonw/datasette/issues/830 MDEyOklzc3VlQ29tbWVudDgxNzQxNDg4MQ== mroswell 192568 2021-04-12T01:06:34Z 2021-04-12T01:07:27Z CONTRIBUTOR

Related: #1285, including arguments for natural breaks, equal interval, etc. modeled after choropleth map legends.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Redesign register_facet_classes plugin hook 636511683  
817403642 https://github.com/simonw/datasette/pull/1296#issuecomment-817403642 https://api.github.com/repos/simonw/datasette/issues/1296 MDEyOklzc3VlQ29tbWVudDgxNzQwMzY0Mg== codecov[bot] 22429695 2021-04-12T00:29:05Z 2021-04-12T00:29:05Z NONE

Codecov Report

Merging #1296 (527a056) into main (0a7621f) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main    #1296   +/-   ##
=======================================
  Coverage   91.51%   91.51%           
=======================================
  Files          34       34           
  Lines        4255     4255           
=======================================
  Hits         3894     3894           
  Misses        361      361           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0a7621f...527a056. Read the comment docs.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Dockerfile: use Ubuntu 20.10 as base 855446829  
817301355 https://github.com/simonw/datasette/issues/1295#issuecomment-817301355 https://api.github.com/repos/simonw/datasette/issues/1295 MDEyOklzc3VlQ29tbWVudDgxNzMwMTM1NQ== simonw 9599 2021-04-11T12:40:25Z 2021-04-11T12:41:06Z OWNER

I could have a page about error codes in the docs, then have https://datasette.io/E123 style URLs for each error core which are shown when that error occurs and redirect to the corresponding documentation section.

Can enforce these with a documentation unit test.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Errors should have links to further information 855296937  
815978405 https://github.com/simonw/datasette/issues/1286#issuecomment-815978405 https://api.github.com/repos/simonw/datasette/issues/1286 MDEyOklzc3VlQ29tbWVudDgxNTk3ODQwNQ== mroswell 192568 2021-04-08T16:47:29Z 2021-04-10T03:59:00Z CONTRIBUTOR

This worked for me:
<td class="col-{{ cell.column|to_css_class }} type-{{ cell.value_type }}">{{ cell.value | replace('", "','; ') | replace('[\"','') | replace('\"]','')}}</td>

I'm sure there is a prettier (and more flexible) way, but for now, this is ever-so-much more pleasant to look at.

------ AFTER:
https://user-images.githubusercontent.com/192568/114062829-f2f20c00-9865-11eb-891d-b0c348a9b433.png">

------ BEFORE:
https://user-images.githubusercontent.com/192568/114062871-01402800-9866-11eb-91ce-91efe4ee45cd.png">

(Note: I didn't figure out how to have one item have no semicolon, while multi-items close with a semicolon, but this is good enough for now. I also didn't figure out how to set up a new jinja filter. I don't want to add to /datasette/utils/init.py as I assume that would get overwritten when upgrading datasette. Having a starter guide on creating jinja filters in datasette would be helpful. (The jinja documentation isn't datasette-specific enough for me to quite nail it.)

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Better default display of arrays of items 849220154  
813480043 https://github.com/simonw/datasette/issues/1293#issuecomment-813480043 https://api.github.com/repos/simonw/datasette/issues/1293 MDEyOklzc3VlQ29tbWVudDgxMzQ4MDA0Mw== simonw 9599 2021-04-05T16:16:17Z 2021-04-05T16:16:17Z OWNER

https://latest.datasette.io/fixtures?sql=explain+select+*+from+paginated_view will be an interesting test query - because paginated_view is defined like this:

CREATE VIEW paginated_view AS
    SELECT
        content,
        '- ' || content || ' -' AS content_extra
    FROM no_primary_key;

So this will help test that the mechanism isn't confused by output columns that are created through a concatenation expression.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Research: Automatically display foreign key links on arbitrary query pages 849978964  
813445512 https://github.com/simonw/datasette/issues/1293#issuecomment-813445512 https://api.github.com/repos/simonw/datasette/issues/1293 MDEyOklzc3VlQ29tbWVudDgxMzQ0NTUxMg== simonw 9599 2021-04-05T15:11:40Z 2021-04-05T15:11:40Z OWNER

Here's some older example code that works with opcodes from Python, in this case to output indexes used by a query: https://github.com/plasticityai/supersqlite/blob/master/supersqlite/idxchk.py

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Research: Automatically display foreign key links on arbitrary query pages 849978964  
813438771 https://github.com/simonw/datasette/issues/1293#issuecomment-813438771 https://api.github.com/repos/simonw/datasette/issues/1293 MDEyOklzc3VlQ29tbWVudDgxMzQzODc3MQ== simonw 9599 2021-04-05T14:58:48Z 2021-04-05T14:58:48Z OWNER

I may need to do something special for rowid columns - there is a RowId opcode that might come into play here.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Research: Automatically display foreign key links on arbitrary query pages 849978964  
813249000 https://github.com/dogsheep/dogsheep-photos/issues/35#issuecomment-813249000 https://api.github.com/repos/dogsheep/dogsheep-photos/issues/35 MDEyOklzc3VlQ29tbWVudDgxMzI0OTAwMA== ligurio 1151557 2021-04-05T07:37:57Z 2021-04-05T07:37:57Z NONE
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Support to annotate photos on other than macOS OSes 842695374  
813167335 https://github.com/simonw/datasette/issues/620#issuecomment-813167335 https://api.github.com/repos/simonw/datasette/issues/620 MDEyOklzc3VlQ29tbWVudDgxMzE2NzMzNQ== simonw 9599 2021-04-05T03:57:22Z 2021-04-05T03:57:22Z OWNER

This may be obsoleted by #1293 - it looks like I may be able to auto-detect these foreign keys for arbitrary queries after all.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Mechanism for indicating foreign key relationships in the table and query page URLs 520667773  
813164282 https://github.com/simonw/datasette/issues/1293#issuecomment-813164282 https://api.github.com/repos/simonw/datasette/issues/1293 MDEyOklzc3VlQ29tbWVudDgxMzE2NDI4Mg== simonw 9599 2021-04-05T03:42:26Z 2021-04-05T03:42:36Z OWNER

Extracting variables with this trick appears to work OK, but you have to pass the correct variables to the explain select... query. Using defaultdict seems to work there:

>>> rows = conn.execute('explain select * from repos where id = :id', defaultdict(int))
>>> [dict(r) for r in rows if r['opcode'] == 'Variable']
[{'addr': 2,
  'opcode': 'Variable',
  'p1': 1,
  'p2': 1,
  'p3': 0,
  'p4': ':id',
  'p5': 0,
  'comment': None}]
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Research: Automatically display foreign key links on arbitrary query pages 849978964  
813162622 https://github.com/simonw/datasette/issues/1293#issuecomment-813162622 https://api.github.com/repos/simonw/datasette/issues/1293 MDEyOklzc3VlQ29tbWVudDgxMzE2MjYyMg== simonw 9599 2021-04-05T03:34:24Z 2021-04-05T03:40:35Z OWNER

This almost works, but throws errors with some queries (anything with a rowid column for example) - it needs a bunch of test coverage.

def columns_for_query(conn, sql):
    rows = conn.execute('explain ' + sql).fetchall()
    table_rootpage_by_register = {r['p1']: r['p2'] for r in rows if r['opcode'] == 'OpenRead'}
    names_by_rootpage = dict(
        conn.execute(
            'select rootpage, name from sqlite_master where rootpage in ({})'.format(
                ', '.join(map(str, table_rootpage_by_register.values()))
            )
        )
    )
    columns_by_column_register = {}
    for row in rows:
        if row['opcode'] == 'Column':
            addr, opcode, table_id, cid, column_register, p4, p5, comment = row
            table = names_by_rootpage[table_rootpage_by_register[table_id]]
            columns_by_column_register[column_register] = (table, cid)
    result_row = [dict(r) for r in rows if r['opcode'] == 'ResultRow'][0]
    registers = list(range(result_row["p1"], result_row["p1"] + result_row["p2"] - 1))
    all_column_names = {}
    for table in names_by_rootpage.values():
        table_xinfo = conn.execute('pragma table_xinfo({})'.format(table)).fetchall()
        for row in table_xinfo:
            all_column_names[(table, row["cid"])] = row["name"]
    final_output = []
    for r in registers:
        try:
            table, cid = columns_by_column_register[r]
            final_output.append((table, all_column_names[table, cid]))
        except KeyError:
            final_output.append((None, None))
    return final_output
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Research: Automatically display foreign key links on arbitrary query pages 849978964  
813134637 https://github.com/simonw/datasette/issues/1293#issuecomment-813134637 https://api.github.com/repos/simonw/datasette/issues/1293 MDEyOklzc3VlQ29tbWVudDgxMzEzNDYzNw== simonw 9599 2021-04-05T01:21:59Z 2021-04-05T01:21:59Z OWNER

http://www.sqlite.org/draft/lang_explain.html says:

Applications should not use EXPLAIN or EXPLAIN QUERY PLAN since their exact behavior is variable and only partially documented.

I'm going to keep exploring this though.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Research: Automatically display foreign key links on arbitrary query pages 849978964  
813134386 https://github.com/simonw/datasette/issues/1293#issuecomment-813134386 https://api.github.com/repos/simonw/datasette/issues/1293 MDEyOklzc3VlQ29tbWVudDgxMzEzNDM4Ng== simonw 9599 2021-04-05T01:20:28Z 2021-04-05T01:20:28Z OWNER

... that output might also provide a better way to extract variables than the current mechanism using a regular expression, by looking for the Variable opcodes.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Research: Automatically display foreign key links on arbitrary query pages 849978964  
813134227 https://github.com/simonw/datasette/issues/1293#issuecomment-813134227 https://api.github.com/repos/simonw/datasette/issues/1293 MDEyOklzc3VlQ29tbWVudDgxMzEzNDIyNw== simonw 9599 2021-04-05T01:19:31Z 2021-04-05T01:19:31Z OWNER
<table> <thead> <tr> <th>addr</th> <th>opcode</th> <th>p1</th> <th>p2</th> <th>p3</th> <th>p4</th> <th>p5</th> <th>comment</th> </tr> </thead> <tbody> <tr> <td>0</td> <td>Init</td> <td>0</td> <td>47</td> <td>0</td> <td></td> <td>00</td> <td></td> </tr> <tr> <td>1</td> <td>OpenRead</td> <td>0</td> <td>51</td> <td>0</td> <td>15</td> <td>00</td> <td></td> </tr> <tr> <td>2</td> <td>Integer</td> <td>15</td> <td>2</td> <td>0</td> <td></td> <td>00</td> <td></td> </tr> <tr> <td>3</td> <td>Once</td> <td>0</td> <td>15</td> <td>0</td> <td></td> <td>00</td> <td></td> </tr> <tr> <td>4</td> <td>OpenEphemeral</td> <td>2</td> <td>1</td> <td>0</td> <td>k(1,)</td> <td>00</td> <td></td> </tr> <tr> <td>5</td> <td>VOpen</td> <td>1</td> <td>0</td> <td>0</td> <td>vtab:3E692C362158</td> <td>00</td> <td></td> </tr> <tr> <td>6</td> <td>String8</td> <td>0</td> <td>5</td> <td>0</td> <td>CPAD_2020a_SuperUnits</td> <td>00</td> <td></td> </tr> <tr> <td>7</td> <td>SCopy</td> <td>7</td> <td>6</td> <td>0</td> <td></td> <td>00</td> <td></td> </tr> <tr> <td>8</td> <td>Integer</td> <td>2</td> <td>3</td> <td>0</td> <td></td> <td>00</td> <td></td> </tr> <tr> <td>9</td> <td>Integer</td> <td>2</td> <td>4</td> <td>0</td> <td></td> <td>00</td> <td></td> </tr> <tr> <td>10</td> <td>VFilter</td> <td>1</td> <td>15</td> <td>3</td> <td></td> <td>00</td> <td></td> </tr> <tr> <td>11</td> <td>Rowid</td> <td>1</td> <td>8</td> <td>0</td> <td></td> <td>00</td> <td></td> </tr> <tr> <td>12</td> <td>MakeRecord</td> <td>8</td> <td>1</td> <td>9</td> <td>C</td> <td>00</td> <td></td> </tr> <tr> <td>13</td> <td>IdxInsert</td> <td>2</td> <td>9</td> <td>8</td> <td>1</td> <td>00</td> <td></td> </tr> <tr> <td>14</td> <td>VNext</td> <td>1</td> <td>11</td> <td>0</td> <td></td> <td>00</td> <td></td> </tr> <tr> <td>15</td> <td>Return</td> <td>2</td> <td>0</td> <td>0</td> <td></td> <td>00</td> <td></td> </tr> <tr> <td>16</td> <td>Rewind</td> <td>2</td> <td>46</td> <td>0</td> <td></td> <td>00</td> <td></td> </tr> <tr> <td>17</td> <td>Column</td> <td>2</td> <td>0</td> <td>1</td> <td></td> <td>00</td> <td></td> </tr> <tr> <td>18</td> <td>IsNull</td> <td>1</td> <td>45</td> <td>0</td> <td></td> <td>00</td> <td></td> </tr> <tr> <td>19</td> <td>SeekRowid</td> <td>0</td> <td>45</td> <td>1</td> <td></td> <td>00</td> <td></td> </tr> <tr> <td>20</td> <td>Column</td> <td>0</td> <td>2</td> <td>11</td> <td></td> <td>00</td> <td></td> </tr> <tr> <td>21</td> <td>Function0</td> <td>1</td> <td>10</td> <td>9</td> <td>like(2)</td> <td>02</td> <td></td> </tr> <tr> <td>22</td> <td>IfNot</td> <td>9</td> <td>45</td> <td>1</td> <td></td> <td>00</td> <td></td> </tr> <tr> <td>23</td> <td>Column</td> <td>0</td> <td>14</td> <td>13</td> <td></td> <td>00</td> <td></td> </tr> <tr> <td>24</td> <td>Function0</td> <td>1</td> <td>12</td> <td>9</td> <td>intersects(2)</td> <td>02</td> <td></td> </tr> <tr> <td>25</td> <td>Ne</td> <td>14</td> <td>45</td> <td>9</td> <td></td> <td>51</td> <td></td> </tr> <tr> <td>26</td> <td>Column</td> <td>0</td> <td>14</td> <td>9</td> <td></td> <td>00</td> <td></td> </tr> <tr> <td>27</td> <td>Function0</td> <td>0</td> <td>9</td> <td>15</td> <td>asgeojson(1)</td> <td>01</td> <td></td> </tr> <tr> <td>28</td> <td>Rowid</td> <td>0</td> <td>16</td> <td>0</td> <td></td> <td>00</td> <td></td> </tr> <tr> <td>29</td> <td>Column</td> <td>0</td> <td>1</td> <td>17</td> <td></td> <td>00</td> <td></td> </tr> <tr> <td>30</td> <td>Column</td> <td>0</td> <td>2</td> <td>18</td> <td></td> <td>00</td> <td></td> </tr> <tr> <td>31</td> <td>Column</td> <td>0</td> <td>3</td> <td>19</td> <td></td> <td>00</td> <td></td> </tr> <tr> <td>32</td> <td>Column</td> <td>0</td> <td>4</td> <td>20</td> <td></td> <td>00</td> <td></td> </tr> <tr> <td>33</td> <td>Column</td> <td>0</td> <td>5</td> <td>21</td> <td></td> <td>00</td> <td></td> </tr> <tr> <td>34</td> <td>Column</td> <td>0</td> <td>6</td> <td>22</td> <td></td> <td>00</td> <td></td> </tr> <tr> <td>35</td> <td>Column</td> <td>0</td> <td>7</td> <td>23</td> <td></td> <td>00</td> <td></td> </tr> <tr> <td>36</td> <td>Column</td> <td>0</td> <td>8</td> <td>24</td> <td></td> <td>00</td> <td></td> </tr> <tr> <td>37</td> <td>Column</td> <td>0</td> <td>9</td> <td>25</td> <td></td> <td>00</td> <td></td> </tr> <tr> <td>38</td> <td>Column</td> <td>0</td> <td>10</td> <td>26</td> <td></td> <td>00</td> <td></td> </tr> <tr> <td>39</td> <td>Column</td> <td>0</td> <td>11</td> <td>27</td> <td></td> <td>00</td> <td></td> </tr> <tr> <td>40</td> <td>RealAffinity</td> <td>27</td> <td>0</td> <td>0</td> <td></td> <td>00</td> <td></td> </tr> <tr> <td>41</td> <td>Column</td> <td>0</td> <td>12</td> <td>28</td> <td></td> <td>00</td> <td></td> </tr> <tr> <td>42</td> <td>Column</td> <td>0</td> <td>13</td> <td>29</td> <td></td> <td>00</td> <td></td> </tr> <tr> <td>43</td> <td>Column</td> <td>0</td> <td>14</td> <td>30</td> <td></td> <td>00</td> <td></td> </tr> <tr> <td>44</td> <td>ResultRow</td> <td>15</td> <td>16</td> <td>0</td> <td></td> <td>00</td> <td></td> </tr> <tr> <td>45</td> <td>Next</td> <td>2</td> <td>17</td> <td>0</td> <td></td> <td>00</td> <td></td> </tr> <tr> <td>46</td> <td>Halt</td> <td>0</td> <td>0</td> <td>0</td> <td></td> <td>00</td> <td></td> </tr> <tr> <td>47</td> <td>Transaction</td> <td>0</td> <td>0</td> <td>265</td> <td>0</td> <td>01</td> <td></td> </tr> <tr> <td>48</td> <td>Variable</td> <td>1</td> <td>31</td> <td>0</td> <td>:freedraw</td> <td>00</td> <td></td> </tr> <tr> <td>49</td> <td>Function0</td> <td>1</td> <td>31</td> <td>7</td> <td>geomfromgeojson(1)</td> <td>01</td> <td></td> </tr> <tr> <td>50</td> <td>String8</td> <td>0</td> <td>10</td> <td>0</td> <td>%mini%</td> <td>00</td> <td></td> </tr> <tr> <td>51</td> <td>Variable</td> <td>1</td> <td>32</td> <td>0</td> <td>:freedraw</td> <td>00</td> <td></td> </tr> <tr> <td>52</td> <td>Function0</td> <td>1</td> <td>32</td> <td>12</td> <td>geomfromgeojson(1)</td> <td>01</td> <td></td> </tr> <tr> <td>53</td> <td>Integer</td> <td>1</td> <td>14</td> <td>0</td> <td></td> <td>00</td> <td></td> </tr> <tr> <td>54</td> <td>Goto</td> <td>0</td> <td>1</td> <td>0</td> <td></td> <td>00</td> <td></td> </tr> </tbody> </table>

Essential documentation for understanding that output: https://www.sqlite.org/opcode.html

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Research: Automatically display foreign key links on arbitrary query pages 849978964  
813134072 https://github.com/simonw/datasette/issues/1293#issuecomment-813134072 https://api.github.com/repos/simonw/datasette/issues/1293 MDEyOklzc3VlQ29tbWVudDgxMzEzNDA3Mg== simonw 9599 2021-04-05T01:18:37Z 2021-04-05T01:18:37Z OWNER

Had a fantastic suggestion on the SQLite forum: it might be possible to get what I want by interpreting the opcodes output by explain select ....

Copying the reply I posted to this thread:

That's really useful, thanks! It looks like it might be possible for me to reconstruct where each column came from using the explain select output.

Here's a complex example: https://calands.datasettes.com/calands?sql=explain+select%0D%0A++AsGeoJSON%28geometry%29%2C+*%0D%0Afrom%0D%0A++CPAD_2020a_SuperUnits%0D%0Awhere%0D%0A++PARK_NAME+like+%27%25mini%25%27+and%0D%0A++Intersects%28GeomFromGeoJSON%28%3Afreedraw%29%2C+geometry%29+%3D+1%0D%0A++and+CPAD_2020a_SuperUnits.rowid+in+%28%0D%0A++++select%0D%0A++++++rowid%0D%0A++++from%0D%0A++++++SpatialIndex%0D%0A++++where%0D%0A++++++f_table_name+%3D+%27CPAD_2020a_SuperUnits%27%0D%0A++++++and+search_frame+%3D+GeomFromGeoJSON%28%3Afreedraw%29%0D%0A++%29&freedraw=%7B%22type%22%3A%22MultiPolygon%22%2C%22coordinates%22%3A%5B%5B%5B%5B-122.42202758789064%2C37.82280243352759%5D%2C%5B-122.39868164062501%2C37.823887203271454%5D%2C%5B-122.38220214843751%2C37.81846319511331%5D%2C%5B-122.35061645507814%2C37.77071473849611%5D%2C%5B-122.34924316406251%2C37.74465712069939%5D%2C%5B-122.37258911132814%2C37.703380457832374%5D%2C%5B-122.39044189453125%2C37.690340943717715%5D%2C%5B-122.41241455078126%2C37.680559803205135%5D%2C%5B-122.44262695312501%2C37.67295135774715%5D%2C%5B-122.47283935546876%2C37.67295135774715%5D%2C%5B-122.52502441406251%2C37.68382032669382%5D%2C%5B-122.53463745117189%2C37.6892542140253%5D%2C%5B-122.54699707031251%2C37.690340943717715%5D%2C%5B-122.55798339843751%2C37.72945260537781%5D%2C%5B-122.54287719726564%2C37.77831314799672%5D%2C%5B-122.49893188476564%2C37.81303878836991%5D%2C%5B-122.46185302734376%2C37.82822612280363%5D%2C%5B-122.42889404296876%2C37.82822612280363%5D%2C%5B-122.42202758789064%2C37.82280243352759%5D%5D%5D%5D%7D

It looks like the opcodes I need to inspect are OpenRead, Column and ResultRow.

OpenRead tells me which tables are being opened - the p2 value (in this case 51) corresponds to the rootpage column in sqlite_master here: https://calands.datasettes.com/calands?sql=select+*+from+sqlite_master - it gets assigned to the register in p1.

The Column opcodes tell me which columns are being read - p1 is that table reference, and p2 is the cid of the column within that table.

The ResultRow opcode then tells me which columns are used in the results. 15 16 means start at the 15th and then read the next 16 columns.

I think this might work!

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Research: Automatically display foreign key links on arbitrary query pages 849978964  
813116177 https://github.com/simonw/datasette/issues/1293#issuecomment-813116177 https://api.github.com/repos/simonw/datasette/issues/1293 MDEyOklzc3VlQ29tbWVudDgxMzExNjE3Nw== simonw 9599 2021-04-04T23:31:00Z 2021-04-04T23:31:00Z OWNER

Sadly it doesn't do what I need. This query should only return one column, but instead I get back every column that was consulted by the query:

https://user-images.githubusercontent.com/9599/113524385-216ca000-9563-11eb-8a7e-cbbe5dfc02f6.png">

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Research: Automatically display foreign key links on arbitrary query pages 849978964  
813115607 https://github.com/simonw/datasette/issues/1293#issuecomment-813115607 https://api.github.com/repos/simonw/datasette/issues/1293 MDEyOklzc3VlQ29tbWVudDgxMzExNTYwNw== simonw 9599 2021-04-04T23:25:15Z 2021-04-04T23:25:15Z OWNER

Oh wow, I just spotted https://github.com/macbre/sql-metadata

Uses tokenized query returned by python-sqlparse and generates query metadata. Extracts column names and tables used by the query. Provides a helper for normalization of SQL queries and tables aliases resolving.

It's for MySQL, PostgreSQL and Hive right now but maybe getting it working with SQLite wouldn't be too hard?

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Research: Automatically display foreign key links on arbitrary query pages 849978964  
813115414 https://github.com/simonw/datasette/issues/1293#issuecomment-813115414 https://api.github.com/repos/simonw/datasette/issues/1293 MDEyOklzc3VlQ29tbWVudDgxMzExNTQxNA== simonw 9599 2021-04-04T23:23:34Z 2021-04-04T23:23:34Z OWNER

The other approach I considered for this was to have my own SQL query parser running in Python, which could pick apart a complex query and figure out which column was sourced from which table. I dropped this idea because it felt that the moment select * came into play a pure parsing approach wouldn't work - I'd need knowledge of the schema in order to resolve the *.

A Python parser approach might be good enough to handle a subset of queries - those that don't use select * for example - and maybe that would be worth shipping? The feature doesn't have to be perfect for it to be useful.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Research: Automatically display foreign key links on arbitrary query pages 849978964  
813114933 https://github.com/simonw/datasette/issues/1293#issuecomment-813114933 https://api.github.com/repos/simonw/datasette/issues/1293 MDEyOklzc3VlQ29tbWVudDgxMzExNDkzMw== simonw 9599 2021-04-04T23:19:22Z 2021-04-04T23:19:22Z OWNER

I asked about this on the SQLite forum: https://sqlite.org/forum/forumpost/0180277fb7

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Research: Automatically display foreign key links on arbitrary query pages 849978964  
813113653 https://github.com/simonw/datasette/issues/1293#issuecomment-813113653 https://api.github.com/repos/simonw/datasette/issues/1293 MDEyOklzc3VlQ29tbWVudDgxMzExMzY1Mw== simonw 9599 2021-04-04T23:10:49Z 2021-04-04T23:10:49Z OWNER

One option I've not fully explored yet: could I write my own custom SQLite C extension which exposes this functionality as a callable function?

Then I could load that extension and run a SQL query something like this:

select database, table, column from analyze_query(:sql_query)

Where analyze_query(...) would be a fancy virtual table function of some sort that uses the underlying sqlite3_column_database_name() C functions to analyze the SQL query and return details of what it would return.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Research: Automatically display foreign key links on arbitrary query pages 849978964  
813113403 https://github.com/simonw/datasette/issues/1293#issuecomment-813113403 https://api.github.com/repos/simonw/datasette/issues/1293 MDEyOklzc3VlQ29tbWVudDgxMzExMzQwMw== simonw 9599 2021-04-04T23:08:48Z 2021-04-04T23:08:48Z OWNER

Worth noting that adding limit 0 to the query still causes it to conduct the permission checks, hopefully while avoiding doing any of the actual work of executing the query:

In [20]: db.execute('select * from compound_primary_key join facetable on facetable.rowid = compound_primary_key.rowid limit 0').fetchall()
    ...: 
args (21, None, None, None, None) kwargs {}
args (20, 'compound_primary_key', 'pk1', 'main', None) kwargs {}
args (20, 'compound_primary_key', 'pk2', 'main', None) kwargs {}
args (20, 'compound_primary_key', 'content', 'main', None) kwargs {}
args (20, 'facetable', 'pk', 'main', None) kwargs {}
args (20, 'facetable', 'created', 'main', None) kwargs {}
args (20, 'facetable', 'planet_int', 'main', None) kwargs {}
args (20, 'facetable', 'on_earth', 'main', None) kwargs {}
args (20, 'facetable', 'state', 'main', None) kwargs {}
args (20, 'facetable', 'city_id', 'main', None) kwargs {}
args (20, 'facetable', 'neighborhood', 'main', None) kwargs {}
args (20, 'facetable', 'tags', 'main', None) kwargs {}
args (20, 'facetable', 'complex_array', 'main', None) kwargs {}
args (20, 'facetable', 'distinct_some_null', 'main', None) kwargs {}
args (20, 'facetable', 'pk', 'main', None) kwargs {}
args (20, 'compound_primary_key', 'ROWID', 'main', None) kwargs {}
Out[20]: []
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Research: Automatically display foreign key links on arbitrary query pages 849978964  
813113218 https://github.com/simonw/datasette/issues/1293#issuecomment-813113218 https://api.github.com/repos/simonw/datasette/issues/1293 MDEyOklzc3VlQ29tbWVudDgxMzExMzIxOA== simonw 9599 2021-04-04T23:07:25Z 2021-04-04T23:07:25Z OWNER

Here are all of the available constants:

In [3]: for k in dir(sqlite3):
   ...:     if k.startswith("SQLITE_"):
   ...:         print(k, getattr(sqlite3, k))
   ...: 
SQLITE_ALTER_TABLE 26
SQLITE_ANALYZE 28
SQLITE_ATTACH 24
SQLITE_CREATE_INDEX 1
SQLITE_CREATE_TABLE 2
SQLITE_CREATE_TEMP_INDEX 3
SQLITE_CREATE_TEMP_TABLE 4
SQLITE_CREATE_TEMP_TRIGGER 5
SQLITE_CREATE_TEMP_VIEW 6
SQLITE_CREATE_TRIGGER 7
SQLITE_CREATE_VIEW 8
SQLITE_CREATE_VTABLE 29
SQLITE_DELETE 9
SQLITE_DENY 1
SQLITE_DETACH 25
SQLITE_DONE 101
SQLITE_DROP_INDEX 10
SQLITE_DROP_TABLE 11
SQLITE_DROP_TEMP_INDEX 12
SQLITE_DROP_TEMP_TABLE 13
SQLITE_DROP_TEMP_TRIGGER 14
SQLITE_DROP_TEMP_VIEW 15
SQLITE_DROP_TRIGGER 16
SQLITE_DROP_VIEW 17
SQLITE_DROP_VTABLE 30
SQLITE_FUNCTION 31
SQLITE_IGNORE 2
SQLITE_INSERT 18
SQLITE_OK 0
SQLITE_PRAGMA 19
SQLITE_READ 20
SQLITE_RECURSIVE 33
SQLITE_REINDEX 27
SQLITE_SAVEPOINT 32
SQLITE_SELECT 21
SQLITE_TRANSACTION 22
SQLITE_UPDATE 23
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Research: Automatically display foreign key links on arbitrary query pages 849978964  
813113175 https://github.com/simonw/datasette/issues/1293#issuecomment-813113175 https://api.github.com/repos/simonw/datasette/issues/1293 MDEyOklzc3VlQ29tbWVudDgxMzExMzE3NQ== simonw 9599 2021-04-04T23:07:01Z 2021-04-04T23:07:01Z OWNER

A more promising route I found involved the db.set_authorizer method. This can be used to log the permission checks that SQLite uses, including checks for permission to access specific columns of specific tables. For a while I thought this could work!

>>> def print_args(*args, **kwargs):
...    print("args", args, "kwargs", kwargs)
...    return sqlite3.SQLITE_OK

>>> db = sqlite3.connect("fixtures.db")
>>> db.execute('select * from compound_primary_key join facetable on rowid').fetchall()
args (21, None, None, None, None) kwargs {}
args (20, 'compound_primary_key', 'pk1', 'main', None) kwargs {}
args (20, 'compound_primary_key', 'pk2', 'main', None) kwargs {}
args (20, 'compound_primary_key', 'content', 'main', None) kwargs {}
args (20, 'facetable', 'pk', 'main', None) kwargs {}
args (20, 'facetable', 'created', 'main', None) kwargs {}
args (20, 'facetable', 'planet_int', 'main', None) kwargs {}
args (20, 'facetable', 'on_earth', 'main', None) kwargs {}
args (20, 'facetable', 'state', 'main', None) kwargs {}
args (20, 'facetable', 'city_id', 'main', None) kwargs {}
args (20, 'facetable', 'neighborhood', 'main', None) kwargs {}
args (20, 'facetable', 'tags', 'main', None) kwargs {}
args (20, 'facetable', 'complex_array', 'main', None) kwargs {}
args (20, 'facetable', 'distinct_some_null', 'main', None) kwargs {}

Those 20 values (where 20 is SQLITE_READ) looked like they were checking permissions for the columns in the order they would be returned!

Then I found a snag:

In [18]: db.execute('select 1 + 1 + (select max(rowid) from facetable)')
args (21, None, None, None, None) kwargs {}
args (31, None, 'max', None, None) kwargs {}
args (20, 'facetable', 'pk', 'main', None) kwargs {}
args (21, None, None, None, None) kwargs {}
args (20, 'facetable', '', None, None) kwargs {}

Once a subselect is involved the order of the 20 checks no longer matches the order in which the columns are returned from the query.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Research: Automatically display foreign key links on arbitrary query pages 849978964  
813112546 https://github.com/simonw/datasette/issues/1293#issuecomment-813112546 https://api.github.com/repos/simonw/datasette/issues/1293 MDEyOklzc3VlQ29tbWVudDgxMzExMjU0Ng== simonw 9599 2021-04-04T23:02:45Z 2021-04-04T23:02:45Z OWNER

I've done various pieces of research into this over the past few years. Capturing what I've discovered in this ticket.

The SQLite C API has functions that can help with this: https://www.sqlite.org/c3ref/column_database_name.html details those. But they're not exposed in the Python SQLite library.

Maybe it would be possible to use them via ctypes? My hunch is that I would have to re-implement the full sqlite3 module with ctypes, which sounds daunting.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Research: Automatically display foreign key links on arbitrary query pages 849978964  
813109789 https://github.com/simonw/datasette/issues/1292#issuecomment-813109789 https://api.github.com/repos/simonw/datasette/issues/1292 MDEyOklzc3VlQ29tbWVudDgxMzEwOTc4OQ== simonw 9599 2021-04-04T22:37:47Z 2021-04-04T22:37:47Z OWNER
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Research ctypes.util.find_library('spatialite') 849975810  
813061516 https://github.com/simonw/datasette/issues/1273#issuecomment-813061516 https://api.github.com/repos/simonw/datasette/issues/1273 MDEyOklzc3VlQ29tbWVudDgxMzA2MTUxNg== simonw 9599 2021-04-04T16:32:40Z 2021-04-04T16:32:40Z OWNER
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Refresh SpatiaLite documentation 838382890  
812941818 https://github.com/simonw/datasette/issues/916#issuecomment-812941818 https://api.github.com/repos/simonw/datasette/issues/916 MDEyOklzc3VlQ29tbWVudDgxMjk0MTgxOA== simonw 9599 2021-04-03T23:43:11Z 2021-04-03T23:43:11Z OWNER

Relevant code is some of the most complex in all of Datasette.

https://github.com/simonw/datasette/blob/0a7621f96f8ad14da17e7172e8a7bce24ef78966/datasette/views/table.py#L530-L594

And

https://github.com/simonw/datasette/blob/0a7621f96f8ad14da17e7172e8a7bce24ef78966/datasette/views/table.py#L743-L771

I'll need to think hard about how to refactor this out into something more understandable before implementing previous links.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Support reverse pagination (previous page, has-previous-items) 672421411  
812941340 https://github.com/simonw/datasette/issues/916#issuecomment-812941340 https://api.github.com/repos/simonw/datasette/issues/916 MDEyOklzc3VlQ29tbWVudDgxMjk0MTM0MA== simonw 9599 2021-04-03T23:38:37Z 2021-04-03T23:38:37Z OWNER
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Support reverse pagination (previous page, has-previous-items) 672421411  
812941112 https://github.com/simonw/datasette/issues/916#issuecomment-812941112 https://api.github.com/repos/simonw/datasette/issues/916 MDEyOklzc3VlQ29tbWVudDgxMjk0MTExMg== simonw 9599 2021-04-03T23:35:55Z 2021-04-03T23:35:55Z OWNER

I tried flipping the direction of the sort and the comparison operators and got this: https://latest.datasette.io/fixtures?sql=select+pk1%2C+pk2%2C+pk3%2C+content+from+compound_three_primary_keys+where+%28%28pk1+%3C+%3Ap0%29%0D%0A++or%0D%0A%28pk1+%3D+%3Ap0+and+pk2+%3C+%3Ap1%29%0D%0A++or%0D%0A%28pk1+%3D+%3Ap0+and+pk2+%3D+%3Ap1+and+pk3+%3C+%3Ap2%29%29+order+by+pk1+desc%2C+pk2+desc%2C+pk3+desc+limit+1+offset+99&p0=a&p1=h&p2=r

select pk1, pk2, pk3, content from compound_three_primary_keys where ((pk1 < :p0)
  or
(pk1 = :p0 and pk2 < :p1)
  or
(pk1 = :p0 and pk2 = :p1 and pk3 < :p2)) order by pk1 desc, pk2 desc, pk3 desc limit 1 offset 99

Which returned a-d-v as desired. I messed around with it to find the limit 1 offset 99 values.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Support reverse pagination (previous page, has-previous-items) 672421411  
812940907 https://github.com/simonw/datasette/issues/916#issuecomment-812940907 https://api.github.com/repos/simonw/datasette/issues/916 MDEyOklzc3VlQ29tbWVudDgxMjk0MDkwNw== simonw 9599 2021-04-03T23:33:41Z 2021-04-03T23:33:41Z OWNER

Let's figure out the SQL for this. The most complex case is probably this one: https://latest.datasette.io/fixtures/compound_three_primary_keys?_next=a%2Ch%2Cr

Here's the SQL for that page: https://latest.datasette.io/fixtures?sql=select+pk1%2C+pk2%2C+pk3%2C+content+from+compound_three_primary_keys+where+%28%28pk1+%3E+%3Ap0%29%0A++or%0A%28pk1+%3D+%3Ap0+and+pk2+%3E+%3Ap1%29%0A++or%0A%28pk1+%3D+%3Ap0+and+pk2+%3D+%3Ap1+and+pk3+%3E+%3Ap2%29%29+order+by+pk1%2C+pk2%2C+pk3+limit+101&p0=a&p1=h&p2=r

select pk1, pk2, pk3, content from compound_three_primary_keys where ((pk1 > :p0)
  or
(pk1 = :p0 and pk2 > :p1)
  or
(pk1 = :p0 and pk2 = :p1 and pk3 > :p2)) order by pk1, pk2, pk3 limit 101

Where p0 is a, p1 is h and p2 is r.

Given the above, how would I figure out the correct previous link? It should be https://latest.datasette.io/fixtures/compound_three_primary_keys?_next=a%2Cd%2Cv - a, d, v.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Support reverse pagination (previous page, has-previous-items) 672421411  
812940457 https://github.com/simonw/datasette/issues/916#issuecomment-812940457 https://api.github.com/repos/simonw/datasette/issues/916 MDEyOklzc3VlQ29tbWVudDgxMjk0MDQ1Nw== simonw 9599 2021-04-03T23:28:40Z 2021-04-03T23:28:40Z OWNER

I think my ideal implementation for this would be to reverse the order, grab the previous page-size-plus-one items, then return a ?_next=x token that would provide the previous page sorted back in the expected default order.

The alternative would be to have a ?_previous=x token which can be used to paginate backwards in reverse order, but I think this would be confusing as it would result in "hit next page, then hit previous page" returning you to a new state which features rows in the reverse order.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Support reverse pagination (previous page, has-previous-items) 672421411  
812935384 https://github.com/simonw/datasette/issues/1287#issuecomment-812935384 https://api.github.com/repos/simonw/datasette/issues/1287 MDEyOklzc3VlQ29tbWVudDgxMjkzNTM4NA== simonw 9599 2021-04-03T22:38:33Z 2021-04-03T22:38:33Z OWNER

https://twitter.com/llanga/status/1378431719934681094 looks like I should wait for 3.9.4, out in a few days.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Upgrade to Python 3.9.4 849396758  
812815358 https://github.com/simonw/datasette/pull/1291#issuecomment-812815358 https://api.github.com/repos/simonw/datasette/issues/1291 MDEyOklzc3VlQ29tbWVudDgxMjgxNTM1OA== codecov[bot] 22429695 2021-04-03T05:32:50Z 2021-04-03T13:31:20Z NONE

Codecov Report

Merging #1291 (73342b6) into main (0a7621f) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main    #1291   +/-   ##
=======================================
  Coverage   91.51%   91.51%           
=======================================
  Files          34       34           
  Lines        4255     4255           
=======================================
  Hits         3894     3894           
  Misses        361      361           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0a7621f...73342b6. Read the comment docs.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Update docs: explain allow_download setting 849582643  
812813732 https://github.com/simonw/datasette/issues/502#issuecomment-812813732 https://api.github.com/repos/simonw/datasette/issues/502 MDEyOklzc3VlQ29tbWVudDgxMjgxMzczMg== louispotok 5413548 2021-04-03T05:16:54Z 2021-04-03T05:16:54Z NONE

For what it's worth, if anyone finds this in the future, I was having the same issue.

After digging through the code, it turned out that the database download is only available if it the db served in immutable mode, so datasette serve -i xyz.db rather than the doc's quickstart recommendation of datasette serve xyz.db.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Exporting sqlite database(s)? 453131917  
812804998 https://github.com/simonw/datasette/issues/916#issuecomment-812804998 https://api.github.com/repos/simonw/datasette/issues/916 MDEyOklzc3VlQ29tbWVudDgxMjgwNDk5OA== simonw 9599 2021-04-03T03:47:45Z 2021-04-03T03:47:45Z OWNER
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Support reverse pagination (previous page, has-previous-items) 672421411  
812804178 https://github.com/simonw/datasette/pull/1290#issuecomment-812804178 https://api.github.com/repos/simonw/datasette/issues/1290 MDEyOklzc3VlQ29tbWVudDgxMjgwNDE3OA== codecov[bot] 22429695 2021-04-03T03:39:16Z 2021-04-03T03:41:29Z NONE

Codecov Report

Merging #1290 (2fb1e42) into main (87b583a) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main    #1290   +/-   ##
=======================================
  Coverage   91.51%   91.51%           
=======================================
  Files          34       34           
  Lines        4255     4255           
=======================================
  Hits         3894     3894           
  Misses        361      361           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 87b583a...2fb1e42. Read the comment docs.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Use pytest-xdist to speed up tests 849568079  
812803256 https://github.com/simonw/datasette/issues/1289#issuecomment-812803256 https://api.github.com/repos/simonw/datasette/issues/1289 MDEyOklzc3VlQ29tbWVudDgxMjgwMzI1Ng== simonw 9599 2021-04-03T03:29:25Z 2021-04-03T03:29:25Z OWNER

https://github.com/simonw/datasette/actions/runs/713207828 ran with pytest-xdist in 4m22s:

https://user-images.githubusercontent.com/9599/113466592-f151bf80-93f1-11eb-9d74-b830e96c19c8.png">

Here's the test suite running on regular pytest in 5m13s:

https://user-images.githubusercontent.com/9599/113466606-00387200-93f2-11eb-95a3-24521c628cbe.png">

Not a huge speed-up because there are only 2 available cores in the GitHub Actions environment, but still worthwhile - especially since this lets people run in parallel on their own laptops.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Speed up tests with pytest-xdist 849543502  
812768915 https://github.com/simonw/datasette/issues/1289#issuecomment-812768915 https://api.github.com/repos/simonw/datasette/issues/1289 MDEyOklzc3VlQ29tbWVudDgxMjc2ODkxNQ== simonw 9599 2021-04-03T00:59:15Z 2021-04-03T00:59:26Z OWNER

Looks like -n auto only detected two cores on GitHub Actions: https://github.com/simonw/datasette/runs/2257597137?check_suite_focus=true

============================= test session starts ==============================
platform linux -- Python 3.7.10, pytest-6.2.2, py-1.10.0, pluggy-0.13.1
SQLite: 3.31.1
rootdir: /home/runner/work/datasette/datasette, configfile: pytest.ini
plugins: xdist-2.2.1, timeout-1.4.2, forked-1.3.0, asyncio-0.14.0
gw0 I / gw1 I
gw0 [878] / gw1 [878]
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Speed up tests with pytest-xdist 849543502  
812767460 https://github.com/simonw/datasette/issues/1289#issuecomment-812767460 https://api.github.com/repos/simonw/datasette/issues/1289 MDEyOklzc3VlQ29tbWVudDgxMjc2NzQ2MA== simonw 9599 2021-04-03T00:48:26Z 2021-04-03T00:48:26Z OWNER

On my Mac pytest-xdist ran the test suite (minus two tests) in 59s, as opposed to 2m23s without xdist.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Speed up tests with pytest-xdist 849543502  
812742462 https://github.com/simonw/datasette/issues/916#issuecomment-812742462 https://api.github.com/repos/simonw/datasette/issues/916 MDEyOklzc3VlQ29tbWVudDgxMjc0MjQ2Mg== jungle-boogie 1111743 2021-04-02T22:37:27Z 2021-04-02T22:37:27Z NONE

Yes, this would be nice!

I using Datasette v0.56 and don't see a previous page button.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Support reverse pagination (previous page, has-previous-items) 672421411  
812711365 https://github.com/simonw/datasette/issues/1245#issuecomment-812711365 https://api.github.com/repos/simonw/datasette/issues/1245 MDEyOklzc3VlQ29tbWVudDgxMjcxMTM2NQ== jungle-boogie 1111743 2021-04-02T20:53:35Z 2021-04-02T20:53:35Z NONE

Yes, I agree.

Alternatively, maybe the header could be at the top and bottom, above the next page button.

Maybe even have the header 50 records down?

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Sticky table column headers would be useful, especially on the query page 817544251  
812710120 https://github.com/simonw/datasette/issues/1255#issuecomment-812710120 https://api.github.com/repos/simonw/datasette/issues/1255 MDEyOklzc3VlQ29tbWVudDgxMjcxMDEyMA== jungle-boogie 1111743 2021-04-02T20:50:08Z 2021-04-02T20:50:08Z NONE

Hello again,

I was able to get my facets running with this settings.json, which was lifted from one of Simon's datasette's and slightly modified.

{
    "default_page_size": 100,
    "max_returned_rows": 1000,
    "num_sql_threads": 3,
    "sql_time_limit_ms": 9000,
    "default_facet_size": 10,
    "facet_time_limit_ms": 9000,
    "facet_suggest_time_limit_ms": 500,
    "hash_urls": false,
    "allow_facet": true,
    "suggest_facets": false,
    "default_cache_ttl": 5,
    "default_cache_ttl_hashed": 31536000,
    "cache_size_kb": 0,
    "allow_csv_stream": true,
    "max_csv_mb": 100,
    "truncate_cells_html": 2048,
    "template_debug": false,
    "base_url": "/"
}
{
    "total_count": 1,
    "+1": 1,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Facets timing out but work when filtering 826700095  
812680519 https://github.com/simonw/datasette/issues/1255#issuecomment-812680519 https://api.github.com/repos/simonw/datasette/issues/1255 MDEyOklzc3VlQ29tbWVudDgxMjY4MDUxOQ== jungle-boogie 1111743 2021-04-02T19:37:57Z 2021-04-02T19:37:57Z NONE

Hello,

I'm also experiencing a timeout in my environment. I don't know if it's because I need more indexes or a more powerful system.

My data has 1,271,111 and when I try to create a facet, there's a time out. I've tried this on two different rows that should significantly filter down data: CITY and PARTY_REG.

Simon's johns_hopkins_csse_daily_reports has more rows and it setup with two facets on load. He does have four indexes created, though. Do I need more indexes?

I have one simple one so far:

CREATE INDEX [idx_party_reg]
    ON [county_active] ([PARTY_REG]);

I'm running Datasette 0.56 installed via pip with Python 3.7.3.

4.19.0-10-amd64 #1 SMP Debian 4.19.132-1 (2020-07-24) x86_64 GNU/Linux

$ cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 10 (buster)"
NAME="Debian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
VERSION_CODENAME=buster
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Facets timing out but work when filtering 826700095  
812679221 https://github.com/simonw/datasette/issues/1286#issuecomment-812679221 https://api.github.com/repos/simonw/datasette/issues/1286 MDEyOklzc3VlQ29tbWVudDgxMjY3OTIyMQ== mroswell 192568 2021-04-02T19:34:01Z 2021-04-02T19:34:01Z CONTRIBUTOR

This shows the city in a different color (and not the comma), but I get the idea, and I like it. (Ooh, could be nice to have the gear have an option in array fields to show as bullets or commas or semicolons...)

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Better default display of arrays of items 849220154  
812665092 https://github.com/simonw/datasette/issues/1287#issuecomment-812665092 https://api.github.com/repos/simonw/datasette/issues/1287 MDEyOklzc3VlQ29tbWVudDgxMjY2NTA5Mg== simonw 9599 2021-04-02T18:54:29Z 2021-04-02T18:54:29Z OWNER

python:3.9.3-slim-buster isn't on Docker Hub yet either.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Upgrade to Python 3.9.4 849396758  
812664443 https://github.com/simonw/datasette/issues/1286#issuecomment-812664443 https://api.github.com/repos/simonw/datasette/issues/1286 MDEyOklzc3VlQ29tbWVudDgxMjY2NDQ0Mw== simonw 9599 2021-04-02T18:52:45Z 2021-04-02T18:52:51Z OWNER

Idea: default to displaying single-dimension JSON arrays of strings as a comma-separated list but show the comma in a different colour - something like this:

https://user-images.githubusercontent.com/9599/113445005-dce9d480-93a9-11eb-901c-0c2fb079c150.png">

I used this HTML for the prototype (re-using .type-int just to get the colour):

<td class="col-tags type-str">tag1<span class="type-int">, </span>tag2</td>
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Better default display of arrays of items 849220154  
812663107 https://github.com/simonw/datasette/issues/1286#issuecomment-812663107 https://api.github.com/repos/simonw/datasette/issues/1286 MDEyOklzc3VlQ29tbWVudDgxMjY2MzEwNw== simonw 9599 2021-04-02T18:49:22Z 2021-04-02T18:49:22Z OWNER

This makes senses - showing an array as ["blah", "blah2", "blah3"] isn't particularly human-friendly!

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Better default display of arrays of items 849220154  
812662583 https://github.com/simonw/datasette/issues/1287#issuecomment-812662583 https://api.github.com/repos/simonw/datasette/issues/1287 MDEyOklzc3VlQ29tbWVudDgxMjY2MjU4Mw== simonw 9599 2021-04-02T18:47:59Z 2021-04-02T18:47:59Z OWNER

Once again having tests for the Dockerfile as seen in #1272 would be useful here.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Upgrade to Python 3.9.4 849396758  
812662026 https://github.com/simonw/datasette/issues/1287#issuecomment-812662026 https://api.github.com/repos/simonw/datasette/issues/1287 MDEyOklzc3VlQ29tbWVudDgxMjY2MjAyNg== simonw 9599 2021-04-02T18:46:20Z 2021-04-02T18:46:20Z OWNER

https://devcenter.heroku.com/articles/python-support#supported-runtimes looks like Heroku still only have 3.9.2 for the moment.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Upgrade to Python 3.9.4 849396758  
812661269 https://github.com/simonw/datasette/issues/1287#issuecomment-812661269 https://api.github.com/repos/simonw/datasette/issues/1287 MDEyOklzc3VlQ29tbWVudDgxMjY2MTI2OQ== simonw 9599 2021-04-02T18:45:08Z 2021-04-02T18:45:19Z OWNER
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Upgrade to Python 3.9.4 849396758  
811362316 https://github.com/dogsheep/dogsheep-photos/pull/31#issuecomment-811362316 https://api.github.com/repos/dogsheep/dogsheep-photos/issues/31 MDEyOklzc3VlQ29tbWVudDgxMTM2MjMxNg== PabloLerma 871250 2021-03-31T19:14:39Z 2021-03-31T19:14:39Z NONE

👋 could I help somehow for this to be merged? As Big Sur is going to be more used as the time goes I think it would be nice to merge and publish a new version. Nice work!

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Update for Big Sur 771511344  
811209922 https://github.com/simonw/datasette/issues/1276#issuecomment-811209922 https://api.github.com/repos/simonw/datasette/issues/1276 MDEyOklzc3VlQ29tbWVudDgxMTIwOTkyMg== justinallen 1314318 2021-03-31T16:27:26Z 2021-03-31T16:27:26Z NONE

Fantastic. Thank you!

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Invalid SQL: "no such table: pragma_database_list" on database page 841456306  
810943882 https://github.com/simonw/datasette/issues/526#issuecomment-810943882 https://api.github.com/repos/simonw/datasette/issues/526 MDEyOklzc3VlQ29tbWVudDgxMDk0Mzg4Mg== jokull 701 2021-03-31T10:03:55Z 2021-03-31T10:03:55Z NONE

+1 on using nested queries to achieve this! Would be great as streaming CSV is an amazing feature.

Some UX/DX details:

I was expecting it to work to simply add &_stream=on to custom SQL queries because the docs say

Any Datasette table, view or custom SQL query can be exported as CSV.

After a bit of testing back and forth I realized streaming only works for full tables.

Would love this feature because I'm using pandas.read_csv to paint graphs from custom queries and the graphs are cut off because of the 1000 row limit.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
CSV streaming for canned queries 459882902  
810779928 https://github.com/simonw/datasette/issues/1284#issuecomment-810779928 https://api.github.com/repos/simonw/datasette/issues/1284 MDEyOklzc3VlQ29tbWVudDgxMDc3OTkyOA== mroswell 192568 2021-03-31T05:40:12Z 2021-03-31T05:40:12Z CONTRIBUTOR

Maybe the addition of two template files: 'one_database_index.html' and 'one_table_index.html' would be a better idea than the documentation diff idea. (They could include commented instructions to rename the preferred template 'index.html', along with any other necessary guidance.)

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Feature or Documentation Request: Individual table as home page template 845794436  
810740486 https://github.com/simonw/datasette/issues/1284#issuecomment-810740486 https://api.github.com/repos/simonw/datasette/issues/1284 MDEyOklzc3VlQ29tbWVudDgxMDc0MDQ4Ng== simonw 9599 2021-03-31T03:57:55Z 2021-03-31T03:57:55Z OWNER

You're right, doing this is really hard at the moment - I'm not sure I know how I would tackle this either, and it's something I've wanted in the past!

I'll have a think about this one.

{
    "total_count": 1,
    "+1": 1,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Feature or Documentation Request: Individual table as home page template 845794436  
809670294 https://github.com/simonw/datasette/pull/1282#issuecomment-809670294 https://api.github.com/repos/simonw/datasette/issues/1282 MDEyOklzc3VlQ29tbWVudDgwOTY3MDI5NA== simonw 9599 2021-03-29T19:57:29Z 2021-03-29T19:57:29Z OWNER

Thanks!

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Fix little typo 843739658  
809667320 https://github.com/simonw/datasette/pull/1282#issuecomment-809667320 https://api.github.com/repos/simonw/datasette/issues/1282 MDEyOklzc3VlQ29tbWVudDgwOTY2NzMyMA== codecov[bot] 22429695 2021-03-29T19:52:35Z 2021-03-29T19:52:35Z NONE

Codecov Report

Merging #1282 (08f7427) into main (0486303) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main    #1282   +/-   ##
=======================================
  Coverage   91.51%   91.51%           
=======================================
  Files          34       34           
  Lines        4255     4255           
=======================================
  Hits         3894     3894           
  Misses        361      361           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0486303...08f7427. Read the comment docs.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Fix little typo 843739658  
809548363 https://github.com/simonw/datasette/issues/696#issuecomment-809548363 https://api.github.com/repos/simonw/datasette/issues/696 MDEyOklzc3VlQ29tbWVudDgwOTU0ODM2Mw== simonw 9599 2021-03-29T17:04:19Z 2021-03-29T17:04:19Z OWNER

I tried this just now against Datasette 0.56 with the new Dockerfile from #1249 (that uses SQLite and SpatiaLite installed with apt-get install) and the tests all passed.

{
    "total_count": 1,
    "+1": 1,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Single failing unit test when run inside the Docker image 576722115  
809010713 https://github.com/simonw/datasette/pull/1031#issuecomment-809010713 https://api.github.com/repos/simonw/datasette/issues/1031 MDEyOklzc3VlQ29tbWVudDgwOTAxMDcxMw== simonw 9599 2021-03-29T01:46:45Z 2021-03-29T01:46:45Z OWNER

Sorry I didn't get to this PR sooner. I've joint-credited you in the release notes for this fix: https://docs.datasette.io/en/stable/changelog.html#v0-56

{
    "total_count": 1,
    "+1": 1,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Fallback to databases in inspect-data.json when no -i options are passed 724369025  
809009580 https://github.com/simonw/datasette/issues/1281#issuecomment-809009580 https://api.github.com/repos/simonw/datasette/issues/1281 MDEyOklzc3VlQ29tbWVudDgwOTAwOTU4MA== simonw 9599 2021-03-29T01:41:48Z 2021-03-29T01:41:48Z OWNER
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Latest Datasette tags missing from Docker Hub 842881221  
809008760 https://github.com/simonw/datasette/issues/1281#issuecomment-809008760 https://api.github.com/repos/simonw/datasette/issues/1281 MDEyOklzc3VlQ29tbWVudDgwOTAwODc2MA== simonw 9599 2021-03-29T01:38:21Z 2021-03-29T01:38:21Z OWNER

Got this error:

"docker tag" requires exactly 2 arguments.
See 'docker tag --help'.

Usage:  docker tag SOURCE_IMAGE[:TAG] TARGET_IMAGE[:TAG]

Create a tag TARGET_IMAGE that refers to SOURCE_IMAGE
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Latest Datasette tags missing from Docker Hub 842881221  
809007255 https://github.com/simonw/datasette/issues/1281#issuecomment-809007255 https://api.github.com/repos/simonw/datasette/issues/1281 MDEyOklzc3VlQ29tbWVudDgwOTAwNzI1NQ== simonw 9599 2021-03-29T01:32:18Z 2021-03-29T01:32:18Z OWNER

I'm going to build a new GitHub Actions workflow for this that lets me manually specify a tag to build and push as a Docker image.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Latest Datasette tags missing from Docker Hub 842881221  
809001653 https://github.com/simonw/datasette/issues/1281#issuecomment-809001653 https://api.github.com/repos/simonw/datasette/issues/1281 MDEyOklzc3VlQ29tbWVudDgwOTAwMTY1Mw== simonw 9599 2021-03-29T01:08:31Z 2021-03-29T01:08:31Z OWNER

I'm going to attempt to fix this manually for the 0.56 release, by building and tagging it by hand and then pushing the 0.56 tag to Docker Hub.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Latest Datasette tags missing from Docker Hub 842881221  
809001273 https://github.com/simonw/datasette/issues/1281#issuecomment-809001273 https://api.github.com/repos/simonw/datasette/issues/1281 MDEyOklzc3VlQ29tbWVudDgwOTAwMTI3Mw== simonw 9599 2021-03-29T01:06:45Z 2021-03-29T01:06:45Z OWNER

https://docs.docker.com/engine/reference/commandline/push/#push-all-tags-of-an-image

Use the -a (or --all-tags) option to push all tags of a local image.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Latest Datasette tags missing from Docker Hub 842881221  
809000903 https://github.com/simonw/datasette/issues/1281#issuecomment-809000903 https://api.github.com/repos/simonw/datasette/issues/1281 MDEyOklzc3VlQ29tbWVudDgwOTAwMDkwMw== simonw 9599 2021-03-29T01:05:10Z 2021-03-29T01:05:10Z OWNER

https://github.com/simonw/datasette/runs/1763835467?check_suite_focus=true for Datasette 0.54 worked, and the output included this:

Successfully tagged ***/datasette:0.54
The push refers to repository [docker.io/***/datasette]
aedd33c6b161: Preparing
...
aedd33c6b161: Pushed
0.54: digest: sha256:65c7e579d1c29755dac5c1ca86b1e97fa88c48bd3d724ac3e02988d0da296140 size: 2005
aedd33c6b161: Preparing
...
5dacd731af1b: Layer already exists
latest: digest: sha256:65c7e579d1c29755dac5c1ca86b1e97fa88c48bd3d724ac3e02988d0da296140 size: 2005

Here's that same section of output from the 0.56 release:

Successfully tagged ***/datasette:0.56
Using default tag: latest
The push refers to repository [docker.io/***/datasette]
4d4a9976adcc: Preparing
...
9b2132a0d5cf: Pushed
latest: digest: sha256:2250d0fbe57b1d615a8d6df0c9d43deb9533532e00bac68854773d8ff8dcf00a size: 1793

The difference here is the "Using default tag: latest" bit.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Latest Datasette tags missing from Docker Hub 842881221  
808999525 https://github.com/simonw/datasette/issues/1281#issuecomment-808999525 https://api.github.com/repos/simonw/datasette/issues/1281 MDEyOklzc3VlQ29tbWVudDgwODk5OTUyNQ== simonw 9599 2021-03-29T01:00:38Z 2021-03-29T01:00:38Z OWNER
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Latest Datasette tags missing from Docker Hub 842881221  
808998719 https://github.com/simonw/datasette/issues/1249#issuecomment-808998719 https://api.github.com/repos/simonw/datasette/issues/1249 MDEyOklzc3VlQ29tbWVudDgwODk5ODcxOQ== simonw 9599 2021-03-29T00:57:13Z 2021-03-29T00:57:13Z OWNER

I just shipped Datasette 0.56 - here's the CI run: https://github.com/simonw/datasette/runs/2214701802?check_suite_focus=true

It pushed a new latest tag to https://hub.docker.com/r/datasetteproject/datasette/tags?page=1&ordering=last_updated

docker pull datasetteproject/datasette:latest

And then:

docker run datasetteproject/datasette:latest datasette \
  --load-extension=spatialite \
  --get /-/versions.json | jq .sqlite

Outputs:

{
  "version": "3.27.2",
  "fts_versions": [
    "FTS5",
    "FTS4",
    "FTS3"
  ],
  "extensions": {
    "json1": null,
    "spatialite": "5.0.1"
  },
  "compile_options": [
    "COMPILER=gcc-8.3.0",
    "ENABLE_COLUMN_METADATA",
    "ENABLE_DBSTAT_VTAB",
    "ENABLE_FTS3",
    "ENABLE_FTS3_PARENTHESIS",
    "ENABLE_FTS3_TOKENIZER",
    "ENABLE_FTS4",
    "ENABLE_FTS5",
    "ENABLE_JSON1",
    "ENABLE_LOAD_EXTENSION",
    "ENABLE_PREUPDATE_HOOK",
    "ENABLE_RTREE",
    "ENABLE_SESSION",
    "ENABLE_STMTVTAB",
    "ENABLE_UNLOCK_NOTIFY",
    "ENABLE_UPDATE_DELETE_LIMIT",
    "HAVE_ISNAN",
    "LIKE_DOESNT_MATCH_BLOBS",
    "MAX_SCHEMA_RETRY=25",
    "MAX_VARIABLE_NUMBER=250000",
    "OMIT_LOOKASIDE",
    "SECURE_DELETE",
    "SOUNDEX",
    "TEMP_STORE=1",
    "THREADSAFE=1",
    "USE_URI"
  ]
}
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Updated Dockerfile with SpatiaLite version 5.0 824064069  
808989067 https://github.com/simonw/datasette/pull/1031#issuecomment-808989067 https://api.github.com/repos/simonw/datasette/issues/1031 MDEyOklzc3VlQ29tbWVudDgwODk4OTA2Nw== simonw 9599 2021-03-29T00:23:41Z 2021-03-29T00:23:41Z OWNER

This bug should have been fixed in #1229 - let me know if that's not the case!

Thanks

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Fallback to databases in inspect-data.json when no -i options are passed 724369025  
808988697 https://github.com/simonw/datasette/pull/1260#issuecomment-808988697 https://api.github.com/repos/simonw/datasette/issues/1260 MDEyOklzc3VlQ29tbWVudDgwODk4ODY5Nw== simonw 9599 2021-03-29T00:22:21Z 2021-03-29T00:22:21Z OWNER

This is interesting!

I've decided to apply a subset of these - the if and elif blocks are a deliberate style choice from me, because I find code clearer when it has if/else as opposed to relying on early termination. Likewise the iteration against .keys() on dictionaries.

I like the other fixes though, I'm about to land them in a separate commit that credits you.

{
    "total_count": 1,
    "+1": 1,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Fix: code quality issues 831163537  
780830464 https://github.com/simonw/datasette/pull/1229#issuecomment-780830464 https://api.github.com/repos/simonw/datasette/issues/1229 MDEyOklzc3VlQ29tbWVudDc4MDgzMDQ2NA== codecov[bot] 22429695 2021-02-17T20:24:30Z 2021-03-29T00:17:21Z NONE

Codecov Report

Merging #1229 (a095248) into main (8e18c79) will not change coverage.
The diff coverage is 100.00%.

@@           Coverage Diff           @@
##             main    #1229   +/-   ##
=======================================
  Coverage   91.51%   91.51%           
=======================================
  Files          34       34           
  Lines        4255     4255           
=======================================
  Hits         3894     3894           
  Misses        361      361           
<table> <thead> <tr> <th>Impacted Files</th> <th>Coverage Δ</th> <th></th> </tr> </thead> <tbody> <tr> <td>datasette/app.py</td> <td>95.85% <100.00%> (ø)</td> <td></td> </tr> </tbody> </table>

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8e18c79...a095248. Read the comment docs.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
ensure immutable databses when starting in configuration directory mode with 810507413  
808987304 https://github.com/simonw/datasette/pull/1229#issuecomment-808987304 https://api.github.com/repos/simonw/datasette/issues/1229 MDEyOklzc3VlQ29tbWVudDgwODk4NzMwNA== simonw 9599 2021-03-29T00:17:13Z 2021-03-29T00:17:13Z OWNER

Thanks for figuring this out!

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
ensure immutable databses when starting in configuration directory mode with 810507413  

Next page

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options:

CREATE TABLE [issue_comments] (
   [html_url] TEXT,
   [issue_url] TEXT,
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [created_at] TEXT,
   [updated_at] TEXT,
   [author_association] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [issue] INTEGER REFERENCES [issues]([id])
, [performed_via_github_app] TEXT);
CREATE INDEX [idx_issue_comments_issue]
                ON [issue_comments] ([issue]);
CREATE INDEX [idx_issue_comments_user]
                ON [issue_comments] ([user]);