247 rows where author_association = "CONTRIBUTOR" sorted by updated_at descending

View and edit SQL

Suggested facets: reactions, created_at (date), updated_at (date)

author_association

  • CONTRIBUTOR · 247
id html_url issue_url node_id user created_at updated_at ▲ author_association body reactions issue performed_via_github_app
876213177 https://github.com/simonw/datasette/issues/1388#issuecomment-876213177 https://api.github.com/repos/simonw/datasette/issues/1388 MDEyOklzc3VlQ29tbWVudDg3NjIxMzE3Nw== aslakr 80737 2021-07-08T07:47:17Z 2021-07-08T07:47:17Z CONTRIBUTOR

This sounds like a valuable feature for people running Datasette behind a proxy.

Yes, in some cases it is easer to use e.g. Apache's ProxyPass Directive with Unix Domain Socket like unix:/home/www.socket|http://localhost/whatever/.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Serve using UNIX domain socket 939051549  
869191854 https://github.com/simonw/datasette/issues/1101#issuecomment-869191854 https://api.github.com/repos/simonw/datasette/issues/1101 MDEyOklzc3VlQ29tbWVudDg2OTE5MTg1NA== eyeseast 25778 2021-06-27T16:42:14Z 2021-06-27T16:42:14Z CONTRIBUTOR

This would really help with this issue: https://github.com/eyeseast/datasette-geojson/issues/7

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
register_output_renderer() should support streaming data 749283032  
869076254 https://github.com/simonw/datasette/issues/1168#issuecomment-869076254 https://api.github.com/repos/simonw/datasette/issues/1168 MDEyOklzc3VlQ29tbWVudDg2OTA3NjI1NA== brandonrobertz 2670795 2021-06-27T00:03:16Z 2021-06-27T00:05:51Z CONTRIBUTOR

Related: Here's an implementation of a get_metadata() plugin hook by @brandonrobertz next-LI@3fd8ce9

Here's a plugin that implements metadata-within-DBs: next-LI/datasette-live-config

How it works: If a database has a __metadata table, then it gets parsed and included in the global metadata. It also implements a database-action hook with a UI for managing config.

More context: https://github.com/next-LI/datasette-live-config/blob/72e335e887f1c69c54c6c2441e07148955b0fc9f/datasette_live_config/__init__.py#L109-L140

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Mechanism for storing metadata in _metadata tables 777333388  
869074701 https://github.com/simonw/datasette/issues/1384#issuecomment-869074701 https://api.github.com/repos/simonw/datasette/issues/1384 MDEyOklzc3VlQ29tbWVudDg2OTA3NDcwMQ== brandonrobertz 2670795 2021-06-26T23:45:18Z 2021-06-26T23:45:37Z CONTRIBUTOR

Here's where the plugin hook is called, demonstrating the fallback= argument:

https://github.com/simonw/datasette/blob/05a312caf3debb51aa1069939923a49e21cd2bd1/datasette/app.py#L426-L472

I'm not convinced of the use-case for passing fallback= to the hook here - is there a reason a plugin might care whether fallback is True or False, seeing as the metadata() method already respects that fallback logic on line 459?

I think you're right. I can't think of a reason why the plugin would care about the fallback parameter since plugins are currently mandated to return a full, global metadata dict.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Plugin hook for dynamic metadata 930807135  
869074182 https://github.com/simonw/datasette/issues/1384#issuecomment-869074182 https://api.github.com/repos/simonw/datasette/issues/1384 MDEyOklzc3VlQ29tbWVudDg2OTA3NDE4Mg== brandonrobertz 2670795 2021-06-26T23:37:42Z 2021-06-26T23:37:42Z CONTRIBUTOR

Hmmm... that's tricky, since one of the most obvious ways to use this hook is to load metadata from database tables using SQL queries.
@brandonrobertz do you have a working example of using this hook to populate metadata from database tables I can try?

Answering my own question: here's how Brandon implements it in his datasette-live-config plugin: https://github.com/next-LI/datasette-live-config/blob/72e335e887f1c69c54c6c2441e07148955b0fc9f/datasette_live_config/__init__.py#L50-L160

That's using a completely separate SQLite connection (actually wrapped in sqlite-utils) and making blocking synchronous calls to it.

This is a pragmatic solution, which works - and likely performs just fine, because SQL queries like this against a small database are so fast that not running them asynchronously isn't actually a problem.

But... it's weird. Everywhere else in Datasette land uses await db.execute(...) - but here's an example where users are encouraged to use blocking calls instead.

Ideally this hook would be asynchronous, but when I started down that path I quickly realized how large of a change this would be, since metadata gets used synchronously across the entire Datasette codebase. (And calling async code from sync is non-trivial.)

In my live-configuration implementation I use synchronous reads using a persistent sqlite connection. This works pretty well in practice, but I agree it's limiting. My thinking around this was to go with the path of least change as Datasette.metadata() is a critical core function.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Plugin hook for dynamic metadata 930807135  
865204472 https://github.com/simonw/datasette/pull/1368#issuecomment-865204472 https://api.github.com/repos/simonw/datasette/issues/1368 MDEyOklzc3VlQ29tbWVudDg2NTIwNDQ3Mg== brandonrobertz 2670795 2021-06-21T17:11:37Z 2021-06-21T17:11:37Z CONTRIBUTOR

If this is a concept ACK then I will move onto fixing the tests (adding new ones) and updating the documentation for the new plugin hook.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
DRAFT: A new plugin hook for dynamic metadata 913865304  
864621099 https://github.com/simonw/sqlite-utils/issues/278#issuecomment-864621099 https://api.github.com/repos/simonw/sqlite-utils/issues/278 MDEyOklzc3VlQ29tbWVudDg2NDYyMTA5OQ== mcint 601708 2021-06-20T22:39:57Z 2021-06-20T22:39:57Z CONTRIBUTOR

Fair. I looked into it, it looks like it could be done, but it would be a bit ugly. I can upload and link a gist of my exploration. Click can parse a first argument while still recognizing it as a sub-command keyword. From there, the program could:
1. ignore it preemptively if it matches a sub-command
2. and/or check if a (db) file exists at the path.

It would then also need to set a shared db argument variable.

Click also makes it easy to parse arguments from environment variables. If you're amenable, I may submit a patch for only that, which would update each sub-command to check for a DB/SQLITE_UTILS_DB environment variable. The goal would be usage that looks like: DB=./convenient.db sqlite-utils [operation] [args]

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Support db as first parameter before subcommand, or as environment variable 923697888  
861497548 https://github.com/simonw/datasette/pull/1130#issuecomment-861497548 https://api.github.com/repos/simonw/datasette/issues/1130 MDEyOklzc3VlQ29tbWVudDg2MTQ5NzU0OA== abdusco 3243482 2021-06-15T13:27:48Z 2021-06-15T13:27:48Z CONTRIBUTOR
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Fix footer not sticking to bottom in short pages 756876238  
857298526 https://github.com/simonw/datasette/pull/1370#issuecomment-857298526 https://api.github.com/repos/simonw/datasette/issues/1370 MDEyOklzc3VlQ29tbWVudDg1NzI5ODUyNg== eyeseast 25778 2021-06-09T01:18:59Z 2021-06-09T01:18:59Z CONTRIBUTOR

I'm happy to grab some or all of these in this PR, if you want.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Ensure db.path is a string before trying to insert into internal database 914130834  
856182547 https://github.com/simonw/datasette/pull/1368#issuecomment-856182547 https://api.github.com/repos/simonw/datasette/issues/1368 MDEyOklzc3VlQ29tbWVudDg1NjE4MjU0Nw== brandonrobertz 2670795 2021-06-07T18:59:47Z 2021-06-07T23:04:25Z CONTRIBUTOR

Note that if we went with a "update_metadata" hook, the hook signature would look something like this (it would return nothing):

update_metadata(
  datasette=self, metadata=metadata, key=key, database=database, table=table,
  fallback=fallback
)

The Datasette function _metadata_recursive_update(self, orig, updated) would disappear into the plugins. Doing this, though, we'd lose the easy ability to make the local metadata.yaml immutable (since we'd no longer have the recursive update).

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
DRAFT: A new plugin hook for dynamic metadata 913865304  
853895159 https://github.com/simonw/datasette/issues/1356#issuecomment-853895159 https://api.github.com/repos/simonw/datasette/issues/1356 MDEyOklzc3VlQ29tbWVudDg1Mzg5NTE1OQ== eyeseast 25778 2021-06-03T14:03:59Z 2021-06-03T14:03:59Z CONTRIBUTOR

(Putting thoughts here to keep the conversation in one place.)

I think using datasette for this use-case is the right approach. I usually have both datasette and sqlite-utils installed in the same project, and that's where I'm trying out queries, so it probably makes the most sense to have datasette also manage the output (and maybe the input, too).

It seems like both --get and --query could work better as subcommands, rather than options, if you're looking at building out a full CLI experience in datasette. It would give a cleaner separation in what you're trying to do and let each have its own dedicated options. So something like this:

# run an arbitrary query
datasette query covid.db "select * from ny_times_us_counties limit 1" --format yaml

# run a canned query
datasette get covid.db some-canned-query --format yaml
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Research: syntactic sugar for using --get with SQL queries, maybe "datasette query" 910092577  
851567204 https://github.com/simonw/datasette/issues/1284#issuecomment-851567204 https://api.github.com/repos/simonw/datasette/issues/1284 MDEyOklzc3VlQ29tbWVudDg1MTU2NzIwNA== mroswell 192568 2021-05-31T15:42:10Z 2021-05-31T15:42:10Z CONTRIBUTOR

I very much want to make:
https://disinfectants.SaferOrToxic.org/disinfectants/listN
have this URL:
https://disinfectants.SaferOrToxic.org/

I'm using only one table page on the site, with no pagination. I'm not using the home page, though when I tried to move my table to the home page as mentioned above, I failed to figure out how.

I am using cloudflare, but I haven't figured out a forwarding or HTML re-write method of doing this, either.

Is there any way I can get a prettier list URL? I'm on Vercel.

(I have a wordpress site on the main domain on a separate host.)

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Feature or Documentation Request: Individual table as home page template 845794436  
850077261 https://github.com/simonw/datasette/pull/1348#issuecomment-850077261 https://api.github.com/repos/simonw/datasette/issues/1348 MDEyOklzc3VlQ29tbWVudDg1MDA3NzI2MQ== blairdrummond 10801138 2021-05-28T03:05:38Z 2021-05-28T03:05:38Z CONTRIBUTOR

Note, the CVEs are probably resolvable with this https://github.com/simonw/datasette/pull/1296 . My experience is that Ubuntu seems to manage these better? Though that is surprising :/

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
DRAFT: add test and scan for docker images 904598267  
846413174 https://github.com/dogsheep/github-to-sqlite/pull/59#issuecomment-846413174 https://api.github.com/repos/dogsheep/github-to-sqlite/issues/59 MDEyOklzc3VlQ29tbWVudDg0NjQxMzE3NA== frosencrantz 631242 2021-05-22T14:06:19Z 2021-05-22T14:06:19Z CONTRIBUTOR

Thanks Simon!

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Remove unneeded exists=True for -a/--auth flag. 771872303  
842798043 https://github.com/simonw/datasette/issues/1236#issuecomment-842798043 https://api.github.com/repos/simonw/datasette/issues/1236 MDEyOklzc3VlQ29tbWVudDg0Mjc5ODA0Mw== mroswell 192568 2021-05-18T03:28:25Z 2021-05-18T03:28:25Z CONTRIBUTOR

That corner handle looks like a hamburger menu to me. Note that the default resize handle is not limited to two-way resize: http://jsfiddle.net/LLrh7Lte/

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Ability to increase size of the SQL editor window 812228314  
838449572 https://github.com/simonw/datasette/pull/1318#issuecomment-838449572 https://api.github.com/repos/simonw/datasette/issues/1318 MDEyOklzc3VlQ29tbWVudDgzODQ0OTU3Mg== dependabot[bot] 49699333 2021-05-11T13:12:30Z 2021-05-11T13:12:30Z CONTRIBUTOR

Superseded by #1321.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Bump black from 21.4b2 to 21.5b0 876431852  
837166862 https://github.com/simonw/datasette/issues/1280#issuecomment-837166862 https://api.github.com/repos/simonw/datasette/issues/1280 MDEyOklzc3VlQ29tbWVudDgzNzE2Njg2Mg== blairdrummond 10801138 2021-05-10T19:07:46Z 2021-05-10T19:07:46Z CONTRIBUTOR

Do you have a list of sqlite versions you want to test against?

One cool thing I saw recently (that we started using) was using import docker within python, and then writing pytest functions which executed against the container

setup

example

The inspiration for this came from the jupyter docker-stacks

So off the top of my head, could look at building the container with different sqlite versions as a build-arg, then run tests against the containers. Just brainstorming though

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Ability to run CI against multiple SQLite versions 842862708  
835491318 https://github.com/simonw/datasette/pull/1296#issuecomment-835491318 https://api.github.com/repos/simonw/datasette/issues/1296 MDEyOklzc3VlQ29tbWVudDgzNTQ5MTMxOA== blairdrummond 10801138 2021-05-08T19:59:01Z 2021-05-08T19:59:01Z CONTRIBUTOR

I have also found that ubuntu has fewer vulnerabilities than the buster based images.

➜  ~ docker pull python:3-buster
➜  ~ trivy image python:3-buster | head                             
2021-04-28T17:14:29.313-0400    INFO    Detecting Debian vulnerabilities...
2021-04-28T17:14:29.393-0400    INFO    Trivy skips scanning programming language libraries because no supported file was detected
python:3-buster (debian 10.9)
=============================
Total: 1621 (UNKNOWN: 13, LOW: 1106, MEDIUM: 343, HIGH: 145, CRITICAL: 14)
+------------------------------+---------------------+----------+------------------------------+---------------+--------------------------------------------------------------+
|           LIBRARY            |  VULNERABILITY ID   | SEVERITY |      INSTALLED VERSION       | FIXED VERSION |                            TITLE                             |
+------------------------------+---------------------+----------+------------------------------+---------------+--------------------------------------------------------------+
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Dockerfile: use Ubuntu 20.10 as base 855446829  
833132571 https://github.com/simonw/datasette/issues/1300#issuecomment-833132571 https://api.github.com/repos/simonw/datasette/issues/1300 MDEyOklzc3VlQ29tbWVudDgzMzEzMjU3MQ== abdusco 3243482 2021-05-06T00:16:50Z 2021-05-06T00:18:05Z CONTRIBUTOR

I ended up using some JS as a workaround.

First, add a JS file in metadata.yaml:

extra_js_urls:
  - '/static/app.js'

then inside the script, find the blob download links and replace .blob extension in the url with .jpg and replace the links with <img/> elements.
You need to add an output formatter to serve BLOB columns as JPG. You can find the code in the first post.
<del>Replacing .blob -> .jpg might not even be necessary, because browsers only care about the mime type, so you only need to serve the binary content with the right content-type header.</del>. You need to replace the extension, otherwise the output renderer will not run.

window.addEventListener('DOMContentLoaded', () => {
    function renderBlobImages() {
        document.querySelectorAll('a[href*=".blob"]').forEach(el => {
            const img = document.createElement('img');
            img.className = 'blob-image';
            img.loading = 'lazy';
            img.src = el.href.replace('.blob', '.jpg');
            el.parentElement.replaceChild(img, el);
        });
    }

    renderBlobImages();
});

while this does the job, I'd prefer handling this in Python where it belongs.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Generating URL for a row inside `render_cell` hook 860625833  
829352402 https://github.com/simonw/datasette/pull/1313#issuecomment-829352402 https://api.github.com/repos/simonw/datasette/issues/1313 MDEyOklzc3VlQ29tbWVudDgyOTM1MjQwMg== dependabot-preview[bot] 27856297 2021-04-29T15:47:23Z 2021-04-29T15:47:23Z CONTRIBUTOR

This pull request will no longer be automatically closed when a new version is found as this pull request was created by Dependabot Preview and this repo is using a version: 2 config file. You can close this pull request and let Dependabot re-create it the next time it checks for updates.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Bump black from 20.8b1 to 21.4b2 871046111  
829260725 https://github.com/simonw/datasette/pull/1311#issuecomment-829260725 https://api.github.com/repos/simonw/datasette/issues/1311 MDEyOklzc3VlQ29tbWVudDgyOTI2MDcyNQ== dependabot-preview[bot] 27856297 2021-04-29T13:58:08Z 2021-04-29T13:58:08Z CONTRIBUTOR

Superseded by #1313.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Bump black from 20.8b1 to 21.4b1 870227815  
828679943 https://github.com/simonw/datasette/pull/1309#issuecomment-828679943 https://api.github.com/repos/simonw/datasette/issues/1309 MDEyOklzc3VlQ29tbWVudDgyODY3OTk0Mw== dependabot-preview[bot] 27856297 2021-04-28T18:26:03Z 2021-04-28T18:26:03Z CONTRIBUTOR

Superseded by #1311.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Bump black from 20.8b1 to 21.4b0 869237023  
823093669 https://github.com/simonw/datasette/issues/1298#issuecomment-823093669 https://api.github.com/repos/simonw/datasette/issues/1298 MDEyOklzc3VlQ29tbWVudDgyMzA5MzY2OQ== mroswell 192568 2021-04-20T08:38:10Z 2021-04-20T08:40:22Z CONTRIBUTOR

@dracos I appreciate your ideas!

  1. Ooh, I like this: https://codepen.io/astro87/pen/LYRQNbd?editors=1100 (That's the codepen from your linked stackoverflow.)
  2. I worry that a max height will be a problem when my facets are open. (I've got 35 active ingredients, and so I've set the default_facet_size to 35.)
  3. I don't understand this one. I'm observing the screenshot... very helpful! (Ah, okay, TR = Top Right and BR = Bottom Right. Absolute grid refers to position style.) All the scroll bars look a little wonky to me. I've also got a lot of facets, and prefer the extra horizontal space so that not as many facets disappear below the fold. My site also has end users... some will be on mobile... not sure what the absolute grid would do there...
  4. (I still think a hover-arrow that scrolls upon click would help, too...)

But meanwhile, I'm going to go ahead and see if I can apply that shadow. (Never would've thought of that.) Hmmm... I'm not an SCSS person. This looks helpful! https://jsonformatter.org/scss-to-css

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
improve table horizontal scroll experience 855476501  
821971059 https://github.com/simonw/datasette/issues/1300#issuecomment-821971059 https://api.github.com/repos/simonw/datasette/issues/1300 MDEyOklzc3VlQ29tbWVudDgyMTk3MTA1OQ== abdusco 3243482 2021-04-18T10:42:19Z 2021-04-18T10:42:19Z CONTRIBUTOR

If there's a simpler way to generate a URL for a specific row, I'm all ears

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Generating URL for a row inside `render_cell` hook 860625833  
821970965 https://github.com/simonw/datasette/issues/1300#issuecomment-821970965 https://api.github.com/repos/simonw/datasette/issues/1300 MDEyOklzc3VlQ29tbWVudDgyMTk3MDk2NQ== abdusco 3243482 2021-04-18T10:41:15Z 2021-04-18T10:41:15Z CONTRIBUTOR

If I change the hookspec and add a row parameter, it works

https://github.com/simonw/datasette/blob/7a2ed9f8a119e220b66d67c7b9e07cbab47b1196/datasette/hookspecs.py#L58

def render_cell(value, column, row, table, database, datasette):

But to generate a URL, I need the primary keys, but I can't call pks = await db.primary_keys(table) inside a sync function. I can't call datasette.utils.detect_primary_keys either, because the db connection is not publicly exposed (AFAICT).

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Generating URL for a row inside `render_cell` hook 860625833  
819467759 https://github.com/simonw/datasette/pull/1296#issuecomment-819467759 https://api.github.com/repos/simonw/datasette/issues/1296 MDEyOklzc3VlQ29tbWVudDgxOTQ2Nzc1OQ== camallen 295329 2021-04-14T12:07:37Z 2021-04-14T12:11:36Z CONTRIBUTOR

Removing /var/lib/apt and /var/lib/dpkg makes apt and dpkg unusable in
images based on this one. Running apt-get clean and removing
/var/lib/apt/lists achieves similar size savings.

this PR helps me as removing the /var/lib/apt and /var/lib/dpkg directories breaks my ability to add packages when using datasetteproject/datasette:0.56 as a base image.


Shorterm workaround for me was to use this in my Dockerfile

FROM datasetteproject/datasette:0.56

RUN mkdir -p /var/lib/apt
RUN mkdir -p /var/lib/dpkg
RUN mkdir -p /var/lib/dpkg/updates
RUN mkdir -p /var/lib/dpkg/info
RUN touch /var/lib/dpkg/status

RUN apt-get update # and install your packages etc
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Dockerfile: use Ubuntu 20.10 as base 855446829  
817414881 https://github.com/simonw/datasette/issues/830#issuecomment-817414881 https://api.github.com/repos/simonw/datasette/issues/830 MDEyOklzc3VlQ29tbWVudDgxNzQxNDg4MQ== mroswell 192568 2021-04-12T01:06:34Z 2021-04-12T01:07:27Z CONTRIBUTOR

Related: #1285, including arguments for natural breaks, equal interval, etc. modeled after choropleth map legends.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Redesign register_facet_classes plugin hook 636511683  
815978405 https://github.com/simonw/datasette/issues/1286#issuecomment-815978405 https://api.github.com/repos/simonw/datasette/issues/1286 MDEyOklzc3VlQ29tbWVudDgxNTk3ODQwNQ== mroswell 192568 2021-04-08T16:47:29Z 2021-04-10T03:59:00Z CONTRIBUTOR

This worked for me:
<td class="col-{{ cell.column|to_css_class }} type-{{ cell.value_type }}">{{ cell.value | replace('", "','; ') | replace('[\"','') | replace('\"]','')}}</td>

I'm sure there is a prettier (and more flexible) way, but for now, this is ever-so-much more pleasant to look at.

------ AFTER:
https://user-images.githubusercontent.com/192568/114062829-f2f20c00-9865-11eb-891d-b0c348a9b433.png">

------ BEFORE:
https://user-images.githubusercontent.com/192568/114062871-01402800-9866-11eb-91ce-91efe4ee45cd.png">

(Note: I didn't figure out how to have one item have no semicolon, while multi-items close with a semicolon, but this is good enough for now. I also didn't figure out how to set up a new jinja filter. I don't want to add to /datasette/utils/init.py as I assume that would get overwritten when upgrading datasette. Having a starter guide on creating jinja filters in datasette would be helpful. (The jinja documentation isn't datasette-specific enough for me to quite nail it.)

{
    "total_count": 1,
    "+1": 1,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Better default display of arrays of items 849220154  
812813732 https://github.com/simonw/datasette/issues/502#issuecomment-812813732 https://api.github.com/repos/simonw/datasette/issues/502 MDEyOklzc3VlQ29tbWVudDgxMjgxMzczMg== louispotok 5413548 2021-04-03T05:16:54Z 2021-04-03T05:16:54Z CONTRIBUTOR

For what it's worth, if anyone finds this in the future, I was having the same issue.

After digging through the code, it turned out that the database download is only available if it the db served in immutable mode, so datasette serve -i xyz.db rather than the doc's quickstart recommendation of datasette serve xyz.db.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Exporting sqlite database(s)? 453131917  
812679221 https://github.com/simonw/datasette/issues/1286#issuecomment-812679221 https://api.github.com/repos/simonw/datasette/issues/1286 MDEyOklzc3VlQ29tbWVudDgxMjY3OTIyMQ== mroswell 192568 2021-04-02T19:34:01Z 2021-04-02T19:34:01Z CONTRIBUTOR

This shows the city in a different color (and not the comma), but I get the idea, and I like it. (Ooh, could be nice to have the gear have an option in array fields to show as bullets or commas or semicolons...)

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Better default display of arrays of items 849220154  
810779928 https://github.com/simonw/datasette/issues/1284#issuecomment-810779928 https://api.github.com/repos/simonw/datasette/issues/1284 MDEyOklzc3VlQ29tbWVudDgxMDc3OTkyOA== mroswell 192568 2021-03-31T05:40:12Z 2021-03-31T05:40:12Z CONTRIBUTOR

Maybe the addition of two template files: 'one_database_index.html' and 'one_table_index.html' would be a better idea than the documentation diff idea. (They could include commented instructions to rename the preferred template 'index.html', along with any other necessary guidance.)

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Feature or Documentation Request: Individual table as home page template 845794436  
805214307 https://github.com/simonw/datasette/issues/1274#issuecomment-805214307 https://api.github.com/repos/simonw/datasette/issues/1274 MDEyOklzc3VlQ29tbWVudDgwNTIxNDMwNw== bobwhitelock 7476523 2021-03-23T20:12:29Z 2021-03-23T20:12:29Z CONTRIBUTOR

One issue I could see with adding first class support for metadata in hjson format is that this would require adding an additional dependency to handle this, for a feature that would be unused by many users. I wonder if this could fit in as a plugin instead; if a hook existed for loading metadata (maybe as part of https://github.com/simonw/datasette/issues/860) the metadata could then come from any source, as specified by plugins, e.g. hjson, toml, XML, a database table etc.

Until/unless this exists, a few ideas for how you could add comments:
- Using YAML as you suggest.
- A common pattern is adding a "comment" key for comments to any object in JSON - I don't think including an unnecessary key like this would break anything in Datasette, but not certain.
- You could use another tool as a preprocessor for your JSON metadata - e.g. hjson or Jsonnet. You'd write the metadata in that format, and then convert that into JSON to actually use as your final metadata.

{
    "total_count": 1,
    "+1": 1,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Might there be some way to comment metadata.json? 839008371  
804640440 https://github.com/simonw/datasette/issues/1153#issuecomment-804640440 https://api.github.com/repos/simonw/datasette/issues/1153 MDEyOklzc3VlQ29tbWVudDgwNDY0MDQ0MA== mroswell 192568 2021-03-23T05:58:20Z 2021-03-23T05:58:20Z CONTRIBUTOR

Could there be a little widget that offers conversion from one to the other?

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Use YAML examples in documentation by default, not JSON 771202454  
804639427 https://github.com/simonw/datasette/pull/1159#issuecomment-804639427 https://api.github.com/repos/simonw/datasette/issues/1159 MDEyOklzc3VlQ29tbWVudDgwNDYzOTQyNw== mroswell 192568 2021-03-23T05:56:02Z 2021-03-23T05:56:02Z CONTRIBUTOR

With just three facets, I like it, but it does take more horizontal space. Would be nice to have a switch somewhere, enabling either original compact option or this proposed more-readable option. Also some control over word wrap (width setting) and facet spacing.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Improve the display of facets information 774332247  
804541064 https://github.com/simonw/datasette/issues/164#issuecomment-804541064 https://api.github.com/repos/simonw/datasette/issues/164 MDEyOklzc3VlQ29tbWVudDgwNDU0MTA2NA== mroswell 192568 2021-03-23T02:45:12Z 2021-03-23T02:45:12Z CONTRIBUTOR

"datasette skeleton" feature removed #476

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
datasette skeleton command for kick-starting database and table metadata 280013907  
804539729 https://github.com/simonw/datasette/issues/163#issuecomment-804539729 https://api.github.com/repos/simonw/datasette/issues/163 MDEyOklzc3VlQ29tbWVudDgwNDUzOTcyOQ== mroswell 192568 2021-03-23T02:41:14Z 2021-03-23T02:41:14Z CONTRIBUTOR

I'm visiting old issues for context while learning datasette. Let me know if okay to make the occasional comment like this one.
querystring argument now located at:
https://docs.datasette.io/en/latest/settings.html#sql-time-limit-ms

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Document the querystring argument for setting a different time limit 279547886  
804471733 https://github.com/simonw/datasette/issues/88#issuecomment-804471733 https://api.github.com/repos/simonw/datasette/issues/88 MDEyOklzc3VlQ29tbWVudDgwNDQ3MTczMw== mroswell 192568 2021-03-22T23:46:36Z 2021-03-22T23:46:36Z CONTRIBUTOR

Google Map API limits seem to prevent https://nhs-england-map.netlify.com from being a working demo.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Add NHS England Hospitals example to wiki 273775212  
804415619 https://github.com/simonw/datasette/issues/1149#issuecomment-804415619 https://api.github.com/repos/simonw/datasette/issues/1149 MDEyOklzc3VlQ29tbWVudDgwNDQxNTYxOQ== mroswell 192568 2021-03-22T21:43:16Z 2021-03-22T21:43:16Z CONTRIBUTOR

Sounds like a good idea.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Make it easier to theme Datasette with CSS 769520939  
803631102 https://github.com/simonw/datasette/issues/942#issuecomment-803631102 https://api.github.com/repos/simonw/datasette/issues/942 MDEyOklzc3VlQ29tbWVudDgwMzYzMTEwMg== mroswell 192568 2021-03-21T17:48:42Z 2021-03-21T17:48:42Z CONTRIBUTOR

I like this idea. Though it might be nice to have some kind of automated system from database to file, so that developers could easily track diffs.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Support column descriptions in metadata.json 681334912  
802923254 https://github.com/simonw/datasette/issues/1265#issuecomment-802923254 https://api.github.com/repos/simonw/datasette/issues/1265 MDEyOklzc3VlQ29tbWVudDgwMjkyMzI1NA== bobwhitelock 7476523 2021-03-19T15:39:15Z 2021-03-19T15:39:15Z CONTRIBUTOR

It doesn't use basic auth, but you can put a whole datasette instance, or parts of this, behind a username/password prompt using https://github.com/simonw/datasette-auth-passwords

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Support for HTTP Basic Authentication 836123030  
802095132 https://github.com/simonw/datasette/issues/1262#issuecomment-802095132 https://api.github.com/repos/simonw/datasette/issues/1262 MDEyOklzc3VlQ29tbWVudDgwMjA5NTEzMg== bobwhitelock 7476523 2021-03-18T16:37:45Z 2021-03-18T16:37:45Z CONTRIBUTOR

This sounds like a good use case for a plugin, since this will only be useful for a subset of Datasette users. It shouldn't be too difficult to add a button to do this with the available plugin hooks - have you taken a look at https://docs.datasette.io/en/latest/writing_plugins.html?

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Plugin hook that could support 'order by random()' for table view 834602299  
799003172 https://github.com/simonw/datasette/issues/236#issuecomment-799003172 https://api.github.com/repos/simonw/datasette/issues/236 MDEyOklzc3VlQ29tbWVudDc5OTAwMzE3Mg== jacobian 21148 2021-03-14T23:42:57Z 2021-03-14T23:42:57Z CONTRIBUTOR

Oh, and the container image can be up to 10GB, so the EFS step might not be needed except for pretty big stuff.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
datasette publish lambda plugin 317001500  
799002993 https://github.com/simonw/datasette/issues/236#issuecomment-799002993 https://api.github.com/repos/simonw/datasette/issues/236 MDEyOklzc3VlQ29tbWVudDc5OTAwMjk5Mw== jacobian 21148 2021-03-14T23:41:51Z 2021-03-14T23:41:51Z CONTRIBUTOR

Now that Lambda supports Docker, this probably is a bit easier and may be able to build on top of the existing package command.

There are weirdnesses in how the command actually gets invoked; the aws-lambda-python image shows a bit of that. So Datasette would probably need some sort of Lambda-specific entry point to make this work.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
datasette publish lambda plugin 317001500  
795112935 https://github.com/simonw/datasette/pull/1256#issuecomment-795112935 https://api.github.com/repos/simonw/datasette/issues/1256 MDEyOklzc3VlQ29tbWVudDc5NTExMjkzNQ== JBPressac 6371750 2021-03-10T08:59:45Z 2021-03-10T08:59:45Z CONTRIBUTOR

Sorry, I meant "minor typo" not "minor type".

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Minor type in IP adress 827341657  
791509910 https://github.com/simonw/datasette/issues/766#issuecomment-791509910 https://api.github.com/repos/simonw/datasette/issues/766 MDEyOklzc3VlQ29tbWVudDc5MTUwOTkxMA== JBPressac 6371750 2021-03-05T15:57:35Z 2021-03-05T16:35:21Z CONTRIBUTOR

Hello,
I have the same wildcards search problems with an instance of Datasette. http://crbc-dataset.huma-num.fr/inventaires/fonds_auguste_dupouy_1872_1967?_search=gwerz&_sort=rowid is OK but http://crbc-dataset.huma-num.fr/inventaires/fonds_auguste_dupouy_1872_1967?_search=gwe* is not (FTS is activated on "Reference" "IntituleAnalyse" "NomDuProducteur" "PresentationDuContenu" "Notes").

Notice that a SQL query as below launched directly from SQLite in the server's shell, retrieves results.

select * from fonds_auguste_dupouy_1872_1967_fts where IntituleAnalyse MATCH "gwe*";

Thanks,

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Enable wildcard-searches by default 617323873  
789186458 https://github.com/simonw/datasette/issues/1238#issuecomment-789186458 https://api.github.com/repos/simonw/datasette/issues/1238 MDEyOklzc3VlQ29tbWVudDc4OTE4NjQ1OA== rgieseke 198537 2021-03-02T20:19:30Z 2021-03-02T20:19:30Z CONTRIBUTOR

A custom templates/index.html seems to work and custom pages as a workaround with moving them to pages/base_url_dir.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Custom pages don't work with base_url setting 813899472  
782430028 https://github.com/simonw/datasette/issues/1212#issuecomment-782430028 https://api.github.com/repos/simonw/datasette/issues/1212 MDEyOklzc3VlQ29tbWVudDc4MjQzMDAyOA== kbaikov 4488943 2021-02-19T22:54:13Z 2021-02-19T22:54:13Z CONTRIBUTOR

I will close this issue since it appears only in my particular setup.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Tests are very slow.  797651831  
782053455 https://github.com/simonw/datasette/pull/1229#issuecomment-782053455 https://api.github.com/repos/simonw/datasette/issues/1229 MDEyOklzc3VlQ29tbWVudDc4MjA1MzQ1NQ== camallen 295329 2021-02-19T12:47:19Z 2021-02-19T12:47:19Z CONTRIBUTOR

I believe this pr and #1031 are related and fix the same issue.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
ensure immutable databses when starting in configuration directory mode with 810507413  
778439617 https://github.com/simonw/datasette/issues/1220#issuecomment-778439617 https://api.github.com/repos/simonw/datasette/issues/1220 MDEyOklzc3VlQ29tbWVudDc3ODQzOTYxNw== bobwhitelock 7476523 2021-02-12T20:33:27Z 2021-02-12T20:33:27Z CONTRIBUTOR

That Docker command will mount your current directory inside the Docker container at /mnt - so you shouldn't need to change anything locally, just run

docker run -p 8001:8001 -v `pwd`:/mnt \
    datasetteproject/datasette \
    datasette -p 8001 -h 0.0.0.0 /mnt/fixtures.db

and it will use the fixtures.db file within your current directory

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Installing datasette via docker: Path 'fixtures.db' does not exist 806743116  
770069864 https://github.com/dogsheep/github-to-sqlite/issues/60#issuecomment-770069864 https://api.github.com/repos/dogsheep/github-to-sqlite/issues/60 MDEyOklzc3VlQ29tbWVudDc3MDA2OTg2NA== daniel-butler 22578954 2021-01-29T21:52:05Z 2021-02-12T18:29:43Z CONTRIBUTOR

For the purposes below I am assuming the organization I would get all the repositories and their related commits from is called gh-organization. The github's owner id of gh-orgnization is 123456789.

github-to-sqlite  repos github.db gh-organization

I'm on a windows computer running git bash to be able to use the | command. This works for me

sqlite3 github.db "SELECT full_name FROM repos WHERE owner = '123456789';" | tr '\n\r' ' ' | xargs | { read repos; github-to-sqlite commits github.db $repos; }

On a pure linux system I think this would work because the new line character is normally \n

sqlite3 github.db "SELECT full_name FROM repos WHERE owner = '123456789';" | tr '\n' ' ' | xargs | { read repos; github-to-sqlite commits github.db $repos; }`

As expected I ran into rate limit issues #51

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Use Data from SQLite in other commands 797097140  
778246347 https://github.com/dogsheep/dogsheep-photos/issues/33#issuecomment-778246347 https://api.github.com/repos/dogsheep/dogsheep-photos/issues/33 MDEyOklzc3VlQ29tbWVudDc3ODI0NjM0Nw== RhetTbull 41546558 2021-02-12T15:00:43Z 2021-02-12T15:00:43Z CONTRIBUTOR

Yes, Big Sur Photos database doesn't have ZGENERICASSET table. PR #31 will fix this.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
photo-to-sqlite: command not found 803338729  
777927946 https://github.com/simonw/datasette/issues/1220#issuecomment-777927946 https://api.github.com/repos/simonw/datasette/issues/1220 MDEyOklzc3VlQ29tbWVudDc3NzkyNzk0Ng== bobwhitelock 7476523 2021-02-12T02:29:54Z 2021-02-12T02:29:54Z CONTRIBUTOR

According to https://github.com/simonw/datasette/blob/master/docs/installation.rst#using-docker it should be

docker run -p 8001:8001 -v `pwd`:/mnt \
    datasetteproject/datasette \
    datasette -p 8001 -h 0.0.0.0 /mnt/fixtures.db

This uses /mnt/fixtures.db whereas you're using fixtures.db - did you try using this path instead?

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Installing datasette via docker: Path 'fixtures.db' does not exist 806743116  
777132761 https://github.com/simonw/datasette/issues/1200#issuecomment-777132761 https://api.github.com/repos/simonw/datasette/issues/1200 MDEyOklzc3VlQ29tbWVudDc3NzEzMjc2MQ== bobwhitelock 7476523 2021-02-11T00:29:52Z 2021-02-11T00:29:52Z CONTRIBUTOR

I'm probably missing something but what's the use case here - what would this offer over adding limit 10 to the query?

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
?_size=10 option for the arbitrary query page would be useful 792890765  
774286962 https://github.com/simonw/datasette/issues/1208#issuecomment-774286962 https://api.github.com/repos/simonw/datasette/issues/1208 MDEyOklzc3VlQ29tbWVudDc3NDI4Njk2Mg== kbaikov 4488943 2021-02-05T21:02:39Z 2021-02-05T21:02:39Z CONTRIBUTOR

@simonw could you please take a look at the PR 1211 that fixes this issue?

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
A lot of open(file) functions are used without a context manager thus producing ResourceWarning: unclosed file <_io.TextIOWrapper 794554881  
772007663 https://github.com/simonw/datasette/issues/1212#issuecomment-772007663 https://api.github.com/repos/simonw/datasette/issues/1212 MDEyOklzc3VlQ29tbWVudDc3MjAwNzY2Mw== kbaikov 4488943 2021-02-02T21:36:56Z 2021-02-02T21:36:56Z CONTRIBUTOR

How do you get 4-5 minutes?
I run my tests in WSL 2, so may be i need to try a real linux VM.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Tests are very slow.  797651831  
771127458 https://github.com/simonw/datasette/pull/1211#issuecomment-771127458 https://api.github.com/repos/simonw/datasette/issues/1211 MDEyOklzc3VlQ29tbWVudDc3MTEyNzQ1OA== kbaikov 4488943 2021-02-01T20:13:39Z 2021-02-01T20:13:39Z CONTRIBUTOR

Ping @simonw

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Use context manager instead of plain open 797649915  
770150526 https://github.com/dogsheep/github-to-sqlite/issues/51#issuecomment-770150526 https://api.github.com/repos/dogsheep/github-to-sqlite/issues/51 MDEyOklzc3VlQ29tbWVudDc3MDE1MDUyNg== daniel-butler 22578954 2021-01-30T03:44:19Z 2021-01-30T03:47:24Z CONTRIBUTOR

I don't have much experience with github's rate limiting. In my day job we use the tenacity library to handle http errors we get.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
github-to-sqlite should handle rate limits better 703246031  
770112248 https://github.com/dogsheep/github-to-sqlite/issues/60#issuecomment-770112248 https://api.github.com/repos/dogsheep/github-to-sqlite/issues/60 MDEyOklzc3VlQ29tbWVudDc3MDExMjI0OA== daniel-butler 22578954 2021-01-30T00:01:03Z 2021-01-30T01:14:42Z CONTRIBUTOR

Yes that would be cool! I wouldn't mind helping. Is this the meat of it? https://github.com/dogsheep/twitter-to-sqlite/blob/21fc1cad6dd6348c67acff90a785b458d3a81275/twitter_to_sqlite/utils.py#L512

It looks like the cli option is added with this decorator : https://github.com/dogsheep/twitter-to-sqlite/blob/21fc1cad6dd6348c67acff90a785b458d3a81275/twitter_to_sqlite/cli.py#L14

I looked a bit at utils.py in the GitHub repository. I was surprised at the amount of manual mapping of the API response you had to do to get this to work.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Use Data from SQLite in other commands 797097140  
754619930 https://github.com/simonw/datasette/issues/1167#issuecomment-754619930 https://api.github.com/repos/simonw/datasette/issues/1167 MDEyOklzc3VlQ29tbWVudDc1NDYxOTkzMA== benpickles 3637 2021-01-05T12:57:57Z 2021-01-05T12:57:57Z CONTRIBUTOR

Not sure where exactly to put the actual docs (presumably somewhere in docs/contributing.rst) but I've made a slight change to make it easier to run locally (copying the approach in excalidraw): https://github.com/simonw/datasette/compare/main...benpickles:prettier-docs

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Add Prettier to contributing documentation 777145954  
647922203 https://github.com/simonw/datasette/issues/859#issuecomment-647922203 https://api.github.com/repos/simonw/datasette/issues/859 MDEyOklzc3VlQ29tbWVudDY0NzkyMjIwMw== abdusco 3243482 2020-06-23T05:44:58Z 2021-01-05T08:22:43Z CONTRIBUTOR

I'm seeing the problem on database page. Index page and table page runs quite fast.

  • Tables have <10 columns (id, url, title, body_html, date, author, meta (for keeping unstructured json)). I've added index on date columns (using sqlite-utils) in addition to the index present on id columns.
  • All tables have FTS enabled on text and varchar columns (title, body_html etc) to speed up searching.
  • There are couple of tables related with foreign keys (think a thread in a forum and posts in that thread, related with thread_id)
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Database page loads too slowly with many large tables (due to table counts) 642572841  
754007242 https://github.com/simonw/datasette/issues/1169#issuecomment-754007242 https://api.github.com/repos/simonw/datasette/issues/1169 MDEyOklzc3VlQ29tbWVudDc1NDAwNzI0Mg== benpickles 3637 2021-01-04T14:29:57Z 2021-01-04T14:29:57Z CONTRIBUTOR

I somewhat share your reluctance to add a package.json to seemingly every project out there but ultimately if they're project dependencies it's important they're managed within the codebase.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Prettier package not actually being cached 777677671  
754004715 https://github.com/simonw/datasette/pull/1170#issuecomment-754004715 https://api.github.com/repos/simonw/datasette/issues/1170 MDEyOklzc3VlQ29tbWVudDc1NDAwNDcxNQ== benpickles 3637 2021-01-04T14:25:44Z 2021-01-04T14:25:44Z CONTRIBUTOR

I was going to re-add the filter to only run Prettier when there have been changes in datasette/static but that would mean it wouldn't run when the package is updated. That plus the fact that the last run of the job took only 8 seconds is why I decided not to re-add the filter.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Install Prettier via package.json 778126516  
753531657 https://github.com/simonw/datasette/issues/1012#issuecomment-753531657 https://api.github.com/repos/simonw/datasette/issues/1012 MDEyOklzc3VlQ29tbWVudDc1MzUzMTY1Nw== bollwyvl 45380 2021-01-02T21:25:36Z 2021-01-02T21:25:36Z CONTRIBUTOR

Actually, on more research, I found out this is handled by the trove-classifiers package now, so it's just a one-liner pr instead of fire-up-a-docker-container-and-do-some-migrations

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
For 1.0 update trove classifier in setup.py 718540751  
752098906 https://github.com/simonw/datasette/issues/417#issuecomment-752098906 https://api.github.com/repos/simonw/datasette/issues/417 MDEyOklzc3VlQ29tbWVudDc1MjA5ODkwNg== psychemedia 82988 2020-12-29T14:34:30Z 2020-12-29T14:34:50Z CONTRIBUTOR

FWIW, I had a look at watchdog for a datasette powered Jupyter notebook search tool: https://github.com/ouseful-testing/nbsearch/blob/main/nbsearch/nbwatchdog.py

Not a production thing, just an experiment trying to explore what might be possible...

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Datasette Library 421546944  
751375487 https://github.com/dogsheep/github-to-sqlite/pull/59#issuecomment-751375487 https://api.github.com/repos/dogsheep/github-to-sqlite/issues/59 MDEyOklzc3VlQ29tbWVudDc1MTM3NTQ4Nw== frosencrantz 631242 2020-12-26T17:08:44Z 2020-12-26T17:08:44Z CONTRIBUTOR

Hi @simonw, do I need to do anything else for this PR to be considered to be included? I've tried using this project and it is quite nice to be able to explore a repository, but noticed that a couple commands don't allow you to use authorization from the environment variable.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Remove unneeded exists=True for -a/--auth flag. 771872303  
750389683 https://github.com/simonw/datasette/pull/1158#issuecomment-750389683 https://api.github.com/repos/simonw/datasette/issues/1158 MDEyOklzc3VlQ29tbWVudDc1MDM4OTY4Mw== eumiro 6774676 2020-12-23T17:02:50Z 2020-12-23T17:02:50Z CONTRIBUTOR

The dict/set suggestion comes from pyupgrade --py36-plus, but then had to black the change.

The rest comes from PyCharm's Inspect code function. I reviewed all the suggestions and fixed a thing or two, such as leading/trailing spaces in the docstrings or turned around the chained conditions.

Then I tried to convert all os.path/glob/open to Path, but there were some local test issues, so I'll have to start over in smaller chunks if you want to have that too.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Modernize code to Python 3.6+ 773913793  
748562330 https://github.com/dogsheep/dogsheep-photos/pull/31#issuecomment-748562330 https://api.github.com/repos/dogsheep/dogsheep-photos/issues/31 MDEyOklzc3VlQ29tbWVudDc0ODU2MjMzMA== RhetTbull 41546558 2020-12-20T04:45:08Z 2020-12-20T04:45:08Z CONTRIBUTOR
{
    "total_count": 1,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 1,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Update for Big Sur 771511344  
748562288 https://github.com/dogsheep/dogsheep-photos/issues/15#issuecomment-748562288 https://api.github.com/repos/dogsheep/dogsheep-photos/issues/15 MDEyOklzc3VlQ29tbWVudDc0ODU2MjI4OA== RhetTbull 41546558 2020-12-20T04:44:22Z 2020-12-20T04:44:22Z CONTRIBUTOR

@nickvazz @simonw I opened a PR that replaces the SQL for ZCOMPUTEDASSETATTRIBUTES to use osxphotos which now exposes all this data and has been updated for Big Sur. I did regression tests to confirm the extracted data is identical, with one exception which should not affect operation: the old code pulled data from ZCOMPUTEDASSETATTRIBUTES for missing photos while the main loop ignores missing photos and does not add them to apple_photos. The new code does not add rows to the apple_photos_scores table for missing photos.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Expose scores from ZCOMPUTEDASSETATTRIBUTES 612151767  
748436779 https://github.com/dogsheep/dogsheep-photos/issues/15#issuecomment-748436779 https://api.github.com/repos/dogsheep/dogsheep-photos/issues/15 MDEyOklzc3VlQ29tbWVudDc0ODQzNjc3OQ== RhetTbull 41546558 2020-12-19T07:49:00Z 2020-12-19T07:49:00Z CONTRIBUTOR

@nickvazz ZGENERICASSET changed to ZASSET in Big Sur. Here's a list of other changes to the schema in Big Sur: https://github.com/RhetTbull/osxphotos/wiki/Changes-in-Photos-6---Big-Sur

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Expose scores from ZCOMPUTEDASSETATTRIBUTES 612151767  
748305976 https://github.com/simonw/datasette/issues/493#issuecomment-748305976 https://api.github.com/repos/simonw/datasette/issues/493 MDEyOklzc3VlQ29tbWVudDc0ODMwNTk3Ng== jefftriplett 50527 2020-12-18T20:34:39Z 2020-12-18T20:34:39Z CONTRIBUTOR

I can't keep up with the renaming contexts, but I like having the ability to run datasette+ datasette-ripgrep against different configs:

datasette serve --metadata=./metadata.json

I have one for all of my code and one per client who has lots of code. So as long as I can point to datasette to something, it's easy to work with.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Rename metadata.json to config.json 449886319  
743080047 https://github.com/simonw/datasette/issues/998#issuecomment-743080047 https://api.github.com/repos/simonw/datasette/issues/998 MDEyOklzc3VlQ29tbWVudDc0MzA4MDA0Nw== JBPressac 6371750 2020-12-11T09:25:09Z 2020-12-11T09:25:09Z CONTRIBUTOR

Hello Simon,
I have a similar problem with horizontal scrollbar display with Datasette version 0.51 and superior for a table with more than 30 rows. With Datasette 0.50, the horizontal scrollbar is displayed, if I upgrade Datasette to 0.51 and superior, the horizontal scrollbar disappears.

Datasette 0.50: horizontal scrollbar

Datasette 0.51 and superior: no horizontal scrollbar

Thanks,

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Wide tables should scroll horizontally within the page 717699884  
738907852 https://github.com/simonw/datasette/pull/1130#issuecomment-738907852 https://api.github.com/repos/simonw/datasette/issues/1130 MDEyOklzc3VlQ29tbWVudDczODkwNzg1Mg== abdusco 3243482 2020-12-04T17:22:29Z 2020-12-04T17:31:25Z CONTRIBUTOR

EDIT: I misunderstood the problem. This seems like a fix better suited for Safari. But I don't have any Apple device to test it.

body {
  min-height: 100vh;
  min-height: -webkit-fill-available;
}
html {
  height: -webkit-fill-available;
}

https://css-tricks.com/css-fix-for-100vh-in-mobile-webkit/


It's actually not that difficult to fix.
Well, this is actually a workaround to keep viewport in place.

I usually put a transition (forgot to do it here) that keeps page from resizing.

.container {
  min-height: 100vh;
  transition: height 10000s steps(0);
}

steps() function prevents excessive layout calculations, and lets the page snap back into place (10000s ~= 3h later) in a single step.
This fix also prevents page from jumping around when the keyboard pops up and down.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Fix footer not sticking to bottom in short pages 756876238  
736322290 https://github.com/simonw/datasette/issues/1111#issuecomment-736322290 https://api.github.com/repos/simonw/datasette/issues/1111 MDEyOklzc3VlQ29tbWVudDczNjMyMjI5MA== abdusco 3243482 2020-12-01T08:54:47Z 2020-12-01T08:54:47Z CONTRIBUTOR

Somewhat related: https://github.com/simonw/datasette/issues/859
I fixed the issue with forking and disabling the counts for hidden tables.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Accessing a database's `.json` is slow for very large SQLite files 751195017  
735281577 https://github.com/simonw/datasette/issues/493#issuecomment-735281577 https://api.github.com/repos/simonw/datasette/issues/493 MDEyOklzc3VlQ29tbWVudDczNTI4MTU3Nw== jefftriplett 50527 2020-11-28T19:39:53Z 2020-11-28T19:39:53Z CONTRIBUTOR

I was confused by --config and I tried passing the json from datasette-ripgrep into config.json just as a wild guess.

A short term solution might be pointing out in plugins that their snippet json can go in metadata.json at least makes it easier to search for config options or to know where to start if someone is new.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Rename metadata.json to config.json 449886319  
735279355 https://github.com/simonw/datasette/pull/1112#issuecomment-735279355 https://api.github.com/repos/simonw/datasette/issues/1112 MDEyOklzc3VlQ29tbWVudDczNTI3OTM1NQ== jefftriplett 50527 2020-11-28T19:21:09Z 2020-11-28T19:21:09Z CONTRIBUTOR

(Even more annoying is that I see my editor leaked an extra delete space at the end of the line. I'm happy to rebuild this to be less annoying, but you probably don't want the changelog update either way)

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Fix --metadata doc usage 752749485  
720354227 https://github.com/simonw/datasette/issues/838#issuecomment-720354227 https://api.github.com/repos/simonw/datasette/issues/838 MDEyOklzc3VlQ29tbWVudDcyMDM1NDIyNw== psychemedia 82988 2020-11-02T09:33:58Z 2020-11-02T09:33:58Z CONTRIBUTOR

Thanks; just a note that the datasette.urls.static(path) and datasette.urls.static_plugins(plugin_name, path) items both seem to be repeated and appear in the docs twice?

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Incorrect URLs when served behind a proxy with base_url set 637395097  
718528252 https://github.com/simonw/datasette/pull/1049#issuecomment-718528252 https://api.github.com/repos/simonw/datasette/issues/1049 MDEyOklzc3VlQ29tbWVudDcxODUyODI1Mg== psychemedia 82988 2020-10-29T09:20:34Z 2020-10-29T09:20:34Z CONTRIBUTOR

That workaround is probably fine. I was trying to work out whether there might be other situations where a pre-external package load might be useful but couldn't offhand bring any other examples to mind. The static plugins option also looks interesting.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Add template block prior to extra URL loaders 729017519  
717359145 https://github.com/simonw/sqlite-utils/pull/189#issuecomment-717359145 https://api.github.com/repos/simonw/sqlite-utils/issues/189 MDEyOklzc3VlQ29tbWVudDcxNzM1OTE0NQ== adamwolf 35681 2020-10-27T16:20:32Z 2020-10-27T16:20:32Z CONTRIBUTOR

No problem. I added a test. Let me know if it looks sufficient or if you want me to to tweak something!

If you don't mind, would you tag this PR as "hacktoberfest-accepted"? If you do mind, no problem and I'm sorry for asking :) My kiddos like the shirts.

{
    "total_count": 1,
    "+1": 1,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Allow iterables other than Lists in m2m records 729818242  
716237524 https://github.com/simonw/datasette/pull/1043#issuecomment-716237524 https://api.github.com/repos/simonw/datasette/issues/1043 MDEyOklzc3VlQ29tbWVudDcxNjIzNzUyNA== bollwyvl 45380 2020-10-26T00:14:57Z 2020-10-26T00:14:57Z CONTRIBUTOR

Sorry, I was out of the loop this weekend. The missing sdists were in some the datasette-* plugins... i'll capture my findings more concretely in one spot when i have a chance...

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Include LICENSE in sdist 727915394  
716123598 https://github.com/simonw/datasette/issues/838#issuecomment-716123598 https://api.github.com/repos/simonw/datasette/issues/838 MDEyOklzc3VlQ29tbWVudDcxNjEyMzU5OA== psychemedia 82988 2020-10-25T10:20:12Z 2020-10-25T10:53:24Z CONTRIBUTOR

I'm trying to run something behind a MyBinder proxy, but seem to have something set up incorrectly and not sure what the fix is?

I'm starting datasette with jupyter-server-proxy setup:

# __init__.py
def setup_nbsearch():

    return {
        "command": [
            "datasette",
            "serve",
            f"{_NBSEARCH_DB_PATH}",
            "-p",
            "{port}",
            "--config",
            "base_url:{base_url}nbsearch/"
        ],
        "absolute_url": True,
        # The following needs a the labextension installing.
        # eg in postBuild: jupyter labextension install jupyterlab-server-proxy
        "launcher_entry": {
            "enabled": True,
            "title": "nbsearch",
        },
    }

where the base_url gets automatically populated by the server-proxy. I define the loaders as:

# __init__.py
from datasette import hookimpl

@hookimpl
def extra_css_urls(database, table, columns, view_name, datasette):
    return [
        "/-/static-plugins/nbsearch/prism.css",
        "/-/static-plugins/nbsearch/nbsearch.css",
    ]

but these seem to also need a base_url prefix set somehow?

Currently, the generated HTML loads properly but internal links are incorrect; eg they take the form <link rel="stylesheet" href="/-/static-plugins/nbsearch/prism.css"> which resolves to eg https://notebooks.gesis.org/hub/-/static-plugins/nbsearch/prism.css rather than required URL of form https://notebooks.gesis.org/binder/jupyter/user/ouseful-testing-nbsearch-0fx1mx67/nbsearch/-/static-plugins/nbsearch/prism.css.

The main css is loaded correctly: <link rel="stylesheet" href="/binder/jupyter/user/ouseful-testing-nbsearch-0fx1mx67/nbsearch/-/static/app.css?404439">

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Incorrect URLs when served behind a proxy with base_url set 637395097  
716066000 https://github.com/simonw/datasette/issues/1033#issuecomment-716066000 https://api.github.com/repos/simonw/datasette/issues/1033 MDEyOklzc3VlQ29tbWVudDcxNjA2NjAwMA== psychemedia 82988 2020-10-24T22:58:33Z 2020-10-24T22:58:33Z CONTRIBUTOR

From the docs, I note:

datasette.urls.instance()
Returns the URL to the Datasette instance root page. This is usually "/"

What about the proxy case? Eg if I am using jupyter-server-proxy on a MyBinder or local Jupyter notebook server site, https://example.com:PORT/weirdpath/datasette, what does datasette.urls.instance() refer to?

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
datasette.urls.static_plugins(...) method 725099777  
714908859 https://github.com/simonw/datasette/issues/1012#issuecomment-714908859 https://api.github.com/repos/simonw/datasette/issues/1012 MDEyOklzc3VlQ29tbWVudDcxNDkwODg1OQ== bollwyvl 45380 2020-10-23T04:49:20Z 2020-10-23T04:49:20Z CONTRIBUTOR

Good luck on 1.0! It may also be worth lobbying for a Framework::Datasette::1.0 classifier. This would be a nice way to allow the ecosystem to self-document a bit more discoverably.

I was surprised to see the PR for Framework::Jupyter is a... database migration! Of course, there may be more workflow to it!

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
For 1.0 update trove classifier in setup.py 718540751  
714657366 https://github.com/simonw/datasette/issues/1033#issuecomment-714657366 https://api.github.com/repos/simonw/datasette/issues/1033 MDEyOklzc3VlQ29tbWVudDcxNDY1NzM2Ng== psychemedia 82988 2020-10-22T17:51:29Z 2020-10-22T17:51:29Z CONTRIBUTOR

How does /-/static relate to current guidance docs around static regarding the --static option and metadata formulations such as "extra_js_urls": [ "/static/app.js"] (I've not managed to get this to work in a Jupyter server proxied set up; the datasette / jupyter server proxy repo may provide a useful test example, eg via MyBinder, for folk to crib from?)

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
datasette.urls.static_plugins(...) method 725099777  
708520800 https://github.com/simonw/datasette/issues/1019#issuecomment-708520800 https://api.github.com/repos/simonw/datasette/issues/1019 MDEyOklzc3VlQ29tbWVudDcwODUyMDgwMA== jsfenfen 639012 2020-10-14T16:37:19Z 2020-10-14T16:37:19Z CONTRIBUTOR

🎉 Thanks so much @simonw ! 🎉

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
"Edit SQL" button on canned queries 721050815  
707326192 https://github.com/dogsheep/swarm-to-sqlite/pull/10#issuecomment-707326192 https://api.github.com/repos/dogsheep/swarm-to-sqlite/issues/10 MDEyOklzc3VlQ29tbWVudDcwNzMyNjE5Mg== mattiaborsoi 29426418 2020-10-12T20:20:02Z 2020-10-12T20:20:02Z CONTRIBUTOR

This closes issue #8

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Update utils.py to fix sqlite3.OperationalError 719637258  
704503719 https://github.com/dogsheep/github-to-sqlite/pull/48#issuecomment-704503719 https://api.github.com/repos/dogsheep/github-to-sqlite/issues/48 MDEyOklzc3VlQ29tbWVudDcwNDUwMzcxOQ== adamjonas 755825 2020-10-06T19:26:59Z 2020-10-06T19:26:59Z CONTRIBUTOR

ref #46

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Add pull requests 681228542  
688573964 https://github.com/simonw/sqlite-utils/pull/146#issuecomment-688573964 https://api.github.com/repos/simonw/sqlite-utils/issues/146 MDEyOklzc3VlQ29tbWVudDY4ODU3Mzk2NA== simonwiles 96218 2020-09-08T01:55:07Z 2020-09-08T01:55:07Z CONTRIBUTOR

Okay, I've rewritten this PR to preserve the batching behaviour but still fix #145, and rebased the branch to account for the db.execute() api change. It's not terribly sophisticated -- if it attempts to insert a batch which has too many variables, the exception is caught, the batch is split in two and each half is inserted separately, and then it carries on as before with the same batch_size. In the edge case where this gets triggered, subsequent batches will all be inserted in two groups too if they continue to have the same number of columns (which is presumably reasonably likely). Do you reckon this is acceptable when set against the awkwardness of recalculating the batch_size on the fly?

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Handle case where subsequent records (after first batch) include extra columns 688668680  
688481317 https://github.com/simonw/sqlite-utils/pull/146#issuecomment-688481317 https://api.github.com/repos/simonw/sqlite-utils/issues/146 MDEyOklzc3VlQ29tbWVudDY4ODQ4MTMxNw== simonwiles 96218 2020-09-07T19:18:55Z 2020-09-07T19:18:55Z CONTRIBUTOR

Just force-pushed to update d042f9c with more formatting changes to satisfy black==20.8b1 and pass the GitHub Actions "Test" workflow.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Handle case where subsequent records (after first batch) include extra columns 688668680  
688479163 https://github.com/simonw/sqlite-utils/pull/146#issuecomment-688479163 https://api.github.com/repos/simonw/sqlite-utils/issues/146 MDEyOklzc3VlQ29tbWVudDY4ODQ3OTE2Mw== simonwiles 96218 2020-09-07T19:10:33Z 2020-09-07T19:11:57Z CONTRIBUTOR

@simonw -- I've gone ahead updated the documentation to reflect the changes introduced in this PR. IMO it's ready to merge now.

In writing the documentation changes, I begin to wonder about the value and role of batch_size at all, tbh. May I assume it was originally intended to prevent using the entire row set to determine columns and column types, and that this was a performance consideration? If so, this PR entirely undermines its purpose. I've been passing in excess of 500,000 rows at a time to insert_all() with these changes and although I'm sure the performance difference is measurable it's not really noticeable; given #145, I don't know that any performance advantages outweigh the problems doing it this way removes. What do you think about just dropping the argument and defaulting to the maximum batch_size permissible given SQLITE_MAX_VARS? Are there other reasons one might want to restrict batch_size that I've overlooked? I could open a new issue to discuss/implement this.

Of course the documentation will need to change again too if/when something is done about #147.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Handle case where subsequent records (after first batch) include extra columns 688668680  
686061028 https://github.com/simonw/datasette/pull/952#issuecomment-686061028 https://api.github.com/repos/simonw/datasette/issues/952 MDEyOklzc3VlQ29tbWVudDY4NjA2MTAyOA== dependabot-preview[bot] 27856297 2020-09-02T22:26:14Z 2020-09-02T22:26:14Z CONTRIBUTOR

Looks like black is up-to-date now, so this is no longer needed.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Update black requirement from ~=19.10b0 to >=19.10,<21.0 687245650  
683382252 https://github.com/simonw/sqlite-utils/issues/145#issuecomment-683382252 https://api.github.com/repos/simonw/sqlite-utils/issues/145 MDEyOklzc3VlQ29tbWVudDY4MzM4MjI1Mg== simonwiles 96218 2020-08-30T06:27:25Z 2020-08-30T06:27:52Z CONTRIBUTOR

Note: had to adjust the test above because trying to exhaust a SQLITE_MAX_VARIABLE_NUMBER of 250000 in 99 records requires 2526 columns, and trips the "Rows can have a maximum of {} columns".format(SQLITE_MAX_VARS) check even before it trips the default SQLITE_MAX_COLUMN value (2000).

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Bug when first record contains fewer columns than subsequent records 688659182  
682815377 https://github.com/simonw/sqlite-utils/issues/139#issuecomment-682815377 https://api.github.com/repos/simonw/sqlite-utils/issues/139 MDEyOklzc3VlQ29tbWVudDY4MjgxNTM3Nw== simonwiles 96218 2020-08-28T16:14:58Z 2020-08-28T16:14:58Z CONTRIBUTOR

Thanks! And yeah, I had updating the docs on my list too :) Will try to get to it this afternoon (budgeting time is fraught with uncertainty at the moment!).

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
insert_all(..., alter=True) should work for new columns introduced after the first 100 records 686978131  
682182178 https://github.com/simonw/sqlite-utils/issues/139#issuecomment-682182178 https://api.github.com/repos/simonw/sqlite-utils/issues/139 MDEyOklzc3VlQ29tbWVudDY4MjE4MjE3OA== simonwiles 96218 2020-08-27T20:46:18Z 2020-08-27T20:46:18Z CONTRIBUTOR

I tried changing the batch_size argument to the total number of records, but it seems only to effect the number of rows that are committed at a time, and has no influence on this problem.

So the reason for this is that the batch_size for import is limited (of necessity) here: https://github.com/simonw/sqlite-utils/blob/main/sqlite_utils/db.py#L1048

With regard to the issue of ignoring columns, however, I made a fork and hacked a temporary fix that looks like this:
https://github.com/simonwiles/sqlite-utils/commit/3901f43c6a712a1a3efc340b5b8d8fd0cbe8ee63

It doesn't seem to affect performance enormously (but I've not tested it thoroughly), and it now does what I need (and would expect, tbh), but it now fails the test here:
https://github.com/simonw/sqlite-utils/blob/main/tests/test_create.py#L710-L716

The existence of this test suggests that insert_all() is behaving as intended, of course. It seems odd to me that this would be a desirable default behaviour (let alone the only behaviour), and its not very prominently flagged-up, either.

@simonw is this something you'd be willing to look at a PR for? I assume you wouldn't want to change the default behaviour at this point, but perhaps an option could be provided, or at least a bit more of a warning in the docs. Are there oversights in the implementation that I've made?

Would be grateful for your thoughts! Thanks!

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
insert_all(..., alter=True) should work for new columns introduced after the first 100 records 686978131  
661524006 https://github.com/simonw/datasette/issues/456#issuecomment-661524006 https://api.github.com/repos/simonw/datasette/issues/456 MDEyOklzc3VlQ29tbWVudDY2MTUyNDAwNg== abeyerpath 32467826 2020-07-21T01:15:07Z 2020-07-21T01:15:07Z CONTRIBUTOR

Bumping this, as the previous fix is passing the wrong type, and not actually addressing the issue...

The exclude argument needs an iterable of packages instead of a single string (but since str is iterable, it's currently excluding packages t, e, and s.)

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Installing installs the tests package 442327592  
655898722 https://github.com/simonw/sqlite-utils/issues/121#issuecomment-655898722 https://api.github.com/repos/simonw/sqlite-utils/issues/121 MDEyOklzc3VlQ29tbWVudDY1NTg5ODcyMg== tsibley 79913 2020-07-09T04:53:08Z 2020-07-09T04:53:08Z CONTRIBUTOR

Yep, I agree that makes more sense for backwards compat and more casual use cases. I think it should be possible for the Database/Queryable methods to DTRT based on seeing if it's within a context-manager-managed transaction.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Improved (and better documented) support for transactions 652961907  
655652679 https://github.com/simonw/sqlite-utils/issues/121#issuecomment-655652679 https://api.github.com/repos/simonw/sqlite-utils/issues/121 MDEyOklzc3VlQ29tbWVudDY1NTY1MjY3OQ== tsibley 79913 2020-07-08T17:24:46Z 2020-07-08T17:24:46Z CONTRIBUTOR

Better transaction handling would be really great. Some of my thoughts on implementing better transaction discipline are in https://github.com/simonw/sqlite-utils/pull/118#issuecomment-655239728.

My preferences:

  • Each CLI command should operate in a single transaction so that either the whole thing succeeds or the whole thing is rolled back. This avoids partially completed operations when an error occurs part way through processing. Partially completed operations are typically much harder to recovery from gracefully and may cause inconsistent data states.

  • The Python API should be transaction-agnostic and rely on the caller to coordinate transactions. Only the caller knows how individual insert, create, update, etc operations/methods should be bundled conceptually into transactions. When the caller is the CLI, for example, that bundling would be at the CLI command-level. Other callers might want to break up operations into multiple transactions. Transactions are usually most useful when controlled at the application-level (like logging configuration) instead of the library level. The library needs to provide an API that's conducive to transaction use, though.

  • The Python API should provide a context manager to provide consistent transactions handling with more useful defaults than Python's sqlite3 module. The latter issues implicit BEGIN statements by default for most DML (INSERT, UPDATE, DELETE, … but not SELECT, I believe), but not DDL (CREATE TABLE, DROP TABLE, CREATE VIEW, …). Notably, the sqlite3 module doesn't issue the implicit BEGIN until the first DML statement. It does not issue it when entering the with conn block, like other DBAPI2-compatible modules do. The with conn block for sqlite3 only arranges to commit or rollback an existing transaction when exiting. Including DDL and SELECTs in transactions is important for operation consistency, though. There are several existing bugs.python.org tickets about this and future changes are in the works, but sqlite-utils can provide its own API sooner. sqlite-utils's Database class could itself be a context manager (built on the sqlite3 connection context manager) which additionally issues an explicit BEGIN when entering. This would then let Python API callers do something like:

db = sqlite_utils.Database(path)

with db: # ← BEGIN issued here by Database.__enter__
    db.insert(…)
    db.create_view(…)
# ← COMMIT/ROLLBACK issue here by sqlite3.connection.__exit__
{
    "total_count": 1,
    "+1": 1,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Improved (and better documented) support for transactions 652961907  
655643078 https://github.com/simonw/sqlite-utils/pull/118#issuecomment-655643078 https://api.github.com/repos/simonw/sqlite-utils/issues/118 MDEyOklzc3VlQ29tbWVudDY1NTY0MzA3OA== tsibley 79913 2020-07-08T17:05:59Z 2020-07-08T17:05:59Z CONTRIBUTOR

The only thing missing from this PR is updates to the documentation.

Ah, yes, thanks for this reminder! I've repushed with doc bits added.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Add insert --truncate option 651844316  
655239728 https://github.com/simonw/sqlite-utils/pull/118#issuecomment-655239728 https://api.github.com/repos/simonw/sqlite-utils/issues/118 MDEyOklzc3VlQ29tbWVudDY1NTIzOTcyOA== tsibley 79913 2020-07-08T02:16:42Z 2020-07-08T02:16:42Z CONTRIBUTOR

I fixed my original oops by moving the DELETE FROM $table out of the chunking loop and repushed. I think this change can be considered in isolation from issues around transactions, which I discuss next.

I wanted to make the DELETE + INSERT happen all in the same transaction so it was robust, but that was more complicated than I expected. The transaction handling in the Database/Table classes isn't systematic, and this poses big hurdles to making Table.insert_all (or other operations) consistent and robust in the face of errors.

For example, I wanted to do this (whitespace ignored in diff, so indentation change not highlighted):

diff --git a/sqlite_utils/db.py b/sqlite_utils/db.py
index d6b9ecf..4107ceb 100644
--- a/sqlite_utils/db.py
+++ b/sqlite_utils/db.py
@@ -1028,6 +1028,11 @@ class Table(Queryable):
         batch_size = max(1, min(batch_size, SQLITE_MAX_VARS // num_columns))
         self.last_rowid = None
         self.last_pk = None
+        with self.db.conn:
+            # Explicit BEGIN is necessary because Python's sqlite3 doesn't
+            # issue implicit BEGINs for DDL, only DML.  We mix DDL and DML
+            # below and might execute DDL first, e.g. for table creation.
+            self.db.conn.execute("BEGIN")
             if truncate and self.exists():
                 self.db.conn.execute("DELETE FROM [{}];".format(self.name))
             for chunk in chunks(itertools.chain([first_record], records), batch_size):
@@ -1038,7 +1043,11 @@ class Table(Queryable):
                         # Use the first batch to derive the table names
                         column_types = suggest_column_types(chunk)
                         column_types.update(columns or {})
-                    self.create(
+                        # Not self.create() because that is wrapped in its own
+                        # transaction and Python's sqlite3 doesn't support
+                        # nested transactions.
+                        self.db.create_table(
+                            self.name,
                             column_types,
                             pk,
                             foreign_keys,
@@ -1139,7 +1148,6 @@ class Table(Queryable):
                     flat_values = list(itertools.chain(*values))
                     queries_and_params = [(sql, flat_values)]

-            with self.db.conn:
                 for query, params in queries_and_params:
                     try:
                         result = self.db.conn.execute(query, params)

but that fails in tests because other methods call insert/upsert/insert_all/upsert_all in the middle of their transactions, so the BEGIN statement throws an error (no nested transactions allowed).

Stepping back, it would be nice to make the transaction handling systematic and predictable. One way to do this is to make the sqlite_utils/db.py code generally not begin or commit any transactions, and require the caller to do that instead. This lets the caller mix and match the Python API calls into transactions as appropriate (which is impossible for the API methods themselves to fully determine). Then, make sqlite_utils/cli.py begin and commit a transaction in each @cli.command function, making each command robust and consistent in the face of errors. The big change here, and why I didn't just submit a patch, is that it dramatically changes the Python API to require callers to begin a transaction rather than just immediately calling methods.

There is also the caveat that for each transaction, an explicit BEGIN is also necessary so that DDL as well as DML (as well as SELECTs) are consistent and rolled back on error. There are several bugs.python.org discussions around this particular problem of DDL and some plans to make it better and consistent with DBAPI2, eventually. In the meantime, the sqlite-utils Database class could be a context manager which supports the incantations necessary to do proper transactions. This would still be a Python API change for callers but wouldn't expose them to the weirdness of the sqlite3's default transaction handling.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Add insert --truncate option 651844316  
655052451 https://github.com/simonw/sqlite-utils/pull/118#issuecomment-655052451 https://api.github.com/repos/simonw/sqlite-utils/issues/118 MDEyOklzc3VlQ29tbWVudDY1NTA1MjQ1MQ== tsibley 79913 2020-07-07T18:45:23Z 2020-07-07T18:45:23Z CONTRIBUTOR

Ah, I see the problem. The truncate is inside a loop I didn't realize was there.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Add insert --truncate option 651844316  
655018966 https://github.com/simonw/sqlite-utils/pull/118#issuecomment-655018966 https://api.github.com/repos/simonw/sqlite-utils/issues/118 MDEyOklzc3VlQ29tbWVudDY1NTAxODk2Ng== tsibley 79913 2020-07-07T17:41:06Z 2020-07-07T17:41:06Z CONTRIBUTOR

Hmm, while tests pass, this may not work as intended on larger datasets. Looking into it.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
Add insert --truncate option 651844316  

Next page

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options:

CREATE TABLE [issue_comments] (
   [html_url] TEXT,
   [issue_url] TEXT,
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [created_at] TEXT,
   [updated_at] TEXT,
   [author_association] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [issue] INTEGER REFERENCES [issues]([id])
, [performed_via_github_app] TEXT);
CREATE INDEX [idx_issue_comments_issue]
                ON [issue_comments] ([issue]);
CREATE INDEX [idx_issue_comments_user]
                ON [issue_comments] ([user]);