issue_comments
10,495 rows sorted by updated_at descending
This data as json, CSV (advanced)
issue >30
- JavaScript plugin hooks mechanism similar to pluggy 47
- Redesign default .json format 46
- Port Datasette to ASGI 42
- Updated Dockerfile with SpatiaLite version 5.0 41
- Authentication (and permissions) as a core concept 40
- await datasette.client.get(path) mechanism for executing internal requests 33
- Maintain an in-memory SQLite table of connected databases and their tables 32
- Ability to sort (and paginate) by column 31
- link_or_copy_directory() error - Invalid cross-device link 28
- Export to CSV 27
- base_url configuration setting 27
- Documentation with recommendations on running Datasette in production without using Docker 27
- Ability for a canned query to write to the database 26
- table.transform() method for advanced alter table 26
- Support cross-database joins 25
- Proof of concept for Datasette on AWS Lambda with EFS 25
- Redesign register_output_renderer callback 24
- "datasette insert" command and plugin hook 23
- Datasette Plugins 22
- .json and .csv exports fail to apply base_url 22
- table.extract(...) method and "sqlite-utils extract" command 21
- Handle spatialite geometry columns better 20
- "flash messages" mechanism 20
- Move CI to GitHub Issues 20
- load_template hook doesn't work for include/extends 20
- ?sort=colname~numeric to sort by by column cast to real 19
- Better way of representing binary data in .csv output 19
- Introspect if table is FTS4 or FTS5 19
- WIP: Add Gmail takeout mbox import 19
- Ability to ship alpha and beta releases 18
- …
reactions 15 ✖
- {"total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} 5,246
- {"total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} 107
- {"total_count": 1, "+1": 0, "-1": 0, "laugh": 0, "hooray": 1, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} 18
- {"total_count": 1, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 1, "rocket": 0, "eyes": 0} 16
- {"total_count": 2, "+1": 2, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} 11
- {"total_count": 1, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 1, "eyes": 0} 6
- {"total_count": 1, "+1": 0, "-1": 0, "laugh": 1, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} 4
- {"total_count": 1, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 1} 3
- {"total_count": 2, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 2, "rocket": 0, "eyes": 0} 3
- {"total_count": 2, "+1": 1, "-1": 0, "laugh": 0, "hooray": 1, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} 3
- {"total_count": 2, "+1": 0, "-1": 0, "laugh": 0, "hooray": 2, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} 2
- {"total_count": 3, "+1": 3, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} 2
- {"total_count": 15, "+1": 7, "-1": 0, "laugh": 1, "hooray": 1, "confused": 0, "heart": 5, "rocket": 1, "eyes": 0} 1
- {"total_count": 2, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 1, "eyes": 0} 1
- {"total_count": 3, "+1": 0, "-1": 0, "laugh": 0, "hooray": 3, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0} 1
id | html_url | issue_url | node_id | user | created_at | updated_at ▲ | author_association | body | reactions | issue | performed_via_github_app |
---|---|---|---|---|---|---|---|---|---|---|---|
804540869 | https://github.com/simonw/datasette/issues/163#issuecomment-804540869 | https://api.github.com/repos/simonw/datasette/issues/163 | MDEyOklzc3VlQ29tbWVudDgwNDU0MDg2OQ== | simonw 9599 | 2021-03-23T02:44:33Z | 2021-03-23T02:44:33Z | OWNER | Comments welcome! |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Document the querystring argument for setting a different time limit 279547886 | |
804539729 | https://github.com/simonw/datasette/issues/163#issuecomment-804539729 | https://api.github.com/repos/simonw/datasette/issues/163 | MDEyOklzc3VlQ29tbWVudDgwNDUzOTcyOQ== | mroswell 192568 | 2021-03-23T02:41:14Z | 2021-03-23T02:41:14Z | CONTRIBUTOR | I'm visiting old issues for context while learning datasette. Let me know if okay to make the occasional comment like this one. querystring argument now located at: https://docs.datasette.io/en/latest/settings.html#sql-time-limit-ms |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Document the querystring argument for setting a different time limit 279547886 | |
804471733 | https://github.com/simonw/datasette/issues/88#issuecomment-804471733 | https://api.github.com/repos/simonw/datasette/issues/88 | MDEyOklzc3VlQ29tbWVudDgwNDQ3MTczMw== | mroswell 192568 | 2021-03-22T23:46:36Z | 2021-03-22T23:46:36Z | CONTRIBUTOR | Google Map API limits seem to prevent https://nhs-england-map.netlify.com from being a working demo. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Add NHS England Hospitals example to wiki 273775212 | |
804299406 | https://github.com/simonw/datasette/pull/1271#issuecomment-804299406 | https://api.github.com/repos/simonw/datasette/issues/1271 | MDEyOklzc3VlQ29tbWVudDgwNDI5OTQwNg== | simonw 9599 | 2021-03-22T18:36:14Z | 2021-03-22T21:49:27Z | OWNER | This isn't actually working - the outer code attempts to send an Need to figure out how to communicate with that thread properly. Also a test that fails in this particular case would be a good idea! |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Use SQLite conn.interrupt() instead of sqlite_timelimit() 837956424 | |
804415619 | https://github.com/simonw/datasette/issues/1149#issuecomment-804415619 | https://api.github.com/repos/simonw/datasette/issues/1149 | MDEyOklzc3VlQ29tbWVudDgwNDQxNTYxOQ== | mroswell 192568 | 2021-03-22T21:43:16Z | 2021-03-22T21:43:16Z | CONTRIBUTOR | Sounds like a good idea. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Make it easier to theme Datasette with CSS 769520939 | |
804406675 | https://github.com/simonw/datasette/issues/1249#issuecomment-804406675 | https://api.github.com/repos/simonw/datasette/issues/1249 | MDEyOklzc3VlQ29tbWVudDgwNDQwNjY3NQ== | simonw 9599 | 2021-03-22T21:26:27Z | 2021-03-22T21:26:27Z | OWNER | (Without the |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Updated Dockerfile with SpatiaLite version 5.0 824064069 | |
804404544 | https://github.com/simonw/datasette/issues/1249#issuecomment-804404544 | https://api.github.com/repos/simonw/datasette/issues/1249 | MDEyOklzc3VlQ29tbWVudDgwNDQwNDU0NA== | simonw 9599 | 2021-03-22T21:22:56Z | 2021-03-22T21:24:24Z | OWNER | Final version of Dockerfile which installs the specified version from GitHub:
```dockerfile FROM python:3.9.2-slim-buster as build Version of Datasette to install, e.g. 0.55docker build . -t datasette --build-arg VERSION=0.55ARG VERSION software-properties-common provides add-apt-repositorywhich we need in order to install a more recent releaseof libsqlite3-mod-spatialite from the sid distributionRUN apt-get update && \ apt-get -y --no-install-recommends install software-properties-common && \ add-apt-repository "deb http://httpredir.debian.org/debian sid main" && \ apt-get update && \ apt-get -t sid install -y --no-install-recommends libsqlite3-mod-spatialite && \ apt-get remove -y software-properties-common && \ apt clean && \ rm -rf /var/lib/apt && \ rm -rf /var/lib/dpkg RUN pip install https://github.com/simonw/datasette/archive/refs/tags/${VERSION}.zip && \ find /usr/local/lib -name 'pycache' | xargs rm -r && \ rm -rf /root/.cache/pip EXPOSE 8001 CMD ["datasette"] ``` Run against 0.55 this produces an image of 262MB |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Updated Dockerfile with SpatiaLite version 5.0 824064069 | |
804384196 | https://github.com/simonw/datasette/issues/1249#issuecomment-804384196 | https://api.github.com/repos/simonw/datasette/issues/1249 | MDEyOklzc3VlQ29tbWVudDgwNDM4NDE5Ng== | simonw 9599 | 2021-03-22T20:48:46Z | 2021-03-22T20:48:46Z | OWNER | I think part of the reason it's smaller is that I ran |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Updated Dockerfile with SpatiaLite version 5.0 824064069 | |
804380181 | https://github.com/simonw/datasette/issues/1249#issuecomment-804380181 | https://api.github.com/repos/simonw/datasette/issues/1249 | MDEyOklzc3VlQ29tbWVudDgwNDM4MDE4MQ== | simonw 9599 | 2021-03-22T20:42:16Z | 2021-03-22T20:42:16Z | OWNER | Considering the image on Docker Hub is 383MB, I'm happy with getting that down to 262MB. I'm going to stop looking for new optimizations here. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Updated Dockerfile with SpatiaLite version 5.0 824064069 | |
804379644 | https://github.com/simonw/datasette/issues/1249#issuecomment-804379644 | https://api.github.com/repos/simonw/datasette/issues/1249 | MDEyOklzc3VlQ29tbWVudDgwNDM3OTY0NA== | simonw 9599 | 2021-03-22T20:41:23Z | 2021-03-22T20:41:23Z | OWNER | I tried adding |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Updated Dockerfile with SpatiaLite version 5.0 824064069 | |
804372977 | https://github.com/simonw/datasette/issues/1249#issuecomment-804372977 | https://api.github.com/repos/simonw/datasette/issues/1249 | MDEyOklzc3VlQ29tbWVudDgwNDM3Mjk3Nw== | simonw 9599 | 2021-03-22T20:30:37Z | 2021-03-22T20:30:37Z | OWNER | I tried copying just the |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Updated Dockerfile with SpatiaLite version 5.0 824064069 | |
804368372 | https://github.com/simonw/datasette/issues/1249#issuecomment-804368372 | https://api.github.com/repos/simonw/datasette/issues/1249 | MDEyOklzc3VlQ29tbWVudDgwNDM2ODM3Mg== | simonw 9599 | 2021-03-22T20:22:43Z | 2021-03-22T20:22:43Z | OWNER | ```dockerfile FROM python:3.9.2-slim-buster as build software-properties-common provides add-apt-repositoryRUN apt-get update && \ apt-get -y --no-install-recommends install software-properties-common && \ add-apt-repository "deb http://httpredir.debian.org/debian sid main" && \ apt-get update && \ apt-get -t sid install -y --no-install-recommends libsqlite3-mod-spatialite && \ apt clean && \ rm -rf /var/lib/apt && \ rm -rf /var/lib/dpkg RUN pip install datasette && \ find /usr/local/lib -name 'pycache' | xargs rm -r && \ rm -rf /root/.cache/pip EXPOSE 8001 CMD ["datasette"] ``` 262 MB |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Updated Dockerfile with SpatiaLite version 5.0 824064069 | |
804363687 | https://github.com/simonw/datasette/issues/1249#issuecomment-804363687 | https://api.github.com/repos/simonw/datasette/issues/1249 | MDEyOklzc3VlQ29tbWVudDgwNDM2MzY4Nw== | simonw 9599 | 2021-03-22T20:15:00Z | 2021-03-22T20:15:00Z | OWNER |
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Updated Dockerfile with SpatiaLite version 5.0 824064069 | |
804360701 | https://github.com/simonw/datasette/issues/1249#issuecomment-804360701 | https://api.github.com/repos/simonw/datasette/issues/1249 | MDEyOklzc3VlQ29tbWVudDgwNDM2MDcwMQ== | simonw 9599 | 2021-03-22T20:10:07Z | 2021-03-22T20:10:07Z | OWNER | Adding |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Updated Dockerfile with SpatiaLite version 5.0 824064069 | |
804347152 | https://github.com/simonw/datasette/issues/1249#issuecomment-804347152 | https://api.github.com/repos/simonw/datasette/issues/1249 | MDEyOklzc3VlQ29tbWVudDgwNDM0NzE1Mg== | simonw 9599 | 2021-03-22T19:47:56Z | 2021-03-22T19:48:03Z | OWNER | I wrote a bunch of tips on creating smaller Docker images here: https://simonwillison.net/2018/Nov/19/smaller-python-docker-images/ |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Updated Dockerfile with SpatiaLite version 5.0 824064069 | |
804344553 | https://github.com/simonw/datasette/issues/1249#issuecomment-804344553 | https://api.github.com/repos/simonw/datasette/issues/1249 | MDEyOklzc3VlQ29tbWVudDgwNDM0NDU1Mw== | simonw 9599 | 2021-03-22T19:43:25Z | 2021-03-22T19:43:25Z | OWNER | Does |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Updated Dockerfile with SpatiaLite version 5.0 824064069 | |
804338678 | https://github.com/simonw/datasette/issues/1249#issuecomment-804338678 | https://api.github.com/repos/simonw/datasette/issues/1249 | MDEyOklzc3VlQ29tbWVudDgwNDMzODY3OA== | simonw 9599 | 2021-03-22T19:33:43Z | 2021-03-22T19:33:43Z | OWNER | Replacing |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Updated Dockerfile with SpatiaLite version 5.0 824064069 | |
804318314 | https://github.com/simonw/datasette/issues/1249#issuecomment-804318314 | https://api.github.com/repos/simonw/datasette/issues/1249 | MDEyOklzc3VlQ29tbWVudDgwNDMxODMxNA== | simonw 9599 | 2021-03-22T19:04:30Z | 2021-03-22T19:04:30Z | OWNER | Considering the image on Docker Hub right now is |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Updated Dockerfile with SpatiaLite version 5.0 824064069 | |
804317545 | https://github.com/simonw/datasette/issues/1249#issuecomment-804317545 | https://api.github.com/repos/simonw/datasette/issues/1249 | MDEyOklzc3VlQ29tbWVudDgwNDMxNzU0NQ== | simonw 9599 | 2021-03-22T19:03:22Z | 2021-03-22T19:03:22Z | OWNER | This Dockerfile: ```dockerfile FROM python:3.9.2-slim-buster as build software-properties-common provides add-apt-repositoryRUN apt-get update && \ apt-get -y install software-properties-common && \ add-apt-repository "deb http://httpredir.debian.org/debian sid main" && \ apt-get update && \ apt-get -t sid install -y libsqlite3-mod-spatialite && \ apt clean && \ rm -rf /var/lib/{apt,dpkg,cache,log}/ RUN pip install datasette EXPOSE 8001 CMD ["datasette"] ``` Produces a 344MB image that includes a working SpatiaLite 5.0 module. And weirdly... it doesn't exhibit the hanging bug! |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Updated Dockerfile with SpatiaLite version 5.0 824064069 | |
804310353 | https://github.com/simonw/datasette/issues/1249#issuecomment-804310353 | https://api.github.com/repos/simonw/datasette/issues/1249 | MDEyOklzc3VlQ29tbWVudDgwNDMxMDM1Mw== | simonw 9599 | 2021-03-22T18:52:12Z | 2021-03-22T18:52:12Z | OWNER | This Dockerfile: ```dockerfile FROM python:3.9.2-slim-buster as build Setup build dependenciesRUN apt update \ && apt install -y python3-dev build-essential wget libxml2-dev libproj-dev \ libminizip-dev libgeos-dev libsqlite3-dev zlib1g-dev pkg-config git \ && apt clean RUN wget "https://www.sqlite.org/2021/sqlite-autoconf-3340100.tar.gz" && tar xzf sqlite-autoconf-3340100.tar.gz \ && cd sqlite-autoconf-3340100 && ./configure --disable-static --enable-fts5 --enable-json1 \ CFLAGS="-g -O2 -DSQLITE_ENABLE_FTS3=1 -DSQLITE_ENABLE_FTS3_PARENTHESIS -DSQLITE_ENABLE_FTS4=1 -DSQLITE_ENABLE_RTREE=1 -DSQLITE_ENABLE_JSON1" \ && make && make install RUN wget "http://www.gaia-gis.it/gaia-sins/freexl-1.0.6.tar.gz" && tar zxf freexl-1.0.6.tar.gz \ && cd freexl-1.0.6 && ./configure && make && make install RUN wget "http://www.gaia-gis.it/gaia-sins/libspatialite-5.0.1.tar.gz" && tar zxf libspatialite-5.0.1.tar.gz \ && cd libspatialite-5.0.1 && ./configure --disable-rttopo && make && make install RUN wget "http://www.gaia-gis.it/gaia-sins/readosm-sources/readosm-1.1.0.tar.gz" && tar zxf readosm-1.1.0.tar.gz && cd readosm-1.1.0 && ./configure && make && make install RUN wget "http://www.gaia-gis.it/gaia-sins/spatialite-tools-5.0.0.tar.gz" && tar zxf spatialite-tools-5.0.0.tar.gz \ && cd spatialite-tools-5.0.0 && ./configure --disable-rttopo && make && make install Add local code to the image instead of fetching from pypi.COPY . /datasetteRUN pip install /datasetteRUN pip install datasette FROM python:3.9.2-slim-buster Copy python dependencies and spatialite librariesCOPY --from=build /usr/local/lib/ /usr/local/lib/ Copy executablesCOPY --from=build /usr/local/bin /usr/local/bin Copy spatial extensionsCOPY --from=build /usr/lib/x86_64-linux-gnu /usr/lib/x86_64-linux-gnu ENV LD_LIBRARY_PATH=/usr/local/lib EXPOSE 8001 CMD ["datasette"] ``` Produced a 448MB image. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Updated Dockerfile with SpatiaLite version 5.0 824064069 | |
804309510 | https://github.com/simonw/datasette/issues/1249#issuecomment-804309510 | https://api.github.com/repos/simonw/datasette/issues/1249 | MDEyOklzc3VlQ29tbWVudDgwNDMwOTUxMA== | simonw 9599 | 2021-03-22T18:50:50Z | 2021-03-22T18:50:50Z | OWNER | Ideally I'd like to use the Debian stable This pattern might let me do that: https://github.com/helmesjo/cpp_bash_utils/blob/f031e926249f8e2d7f260f22dc8974c6d5be11fe/docker/images/linux-gcc.dockerfile#L20-L24 |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Updated Dockerfile with SpatiaLite version 5.0 824064069 | |
804265042 | https://github.com/simonw/datasette/pull/1271#issuecomment-804265042 | https://api.github.com/repos/simonw/datasette/issues/1271 | MDEyOklzc3VlQ29tbWVudDgwNDI2NTA0Mg== | simonw 9599 | 2021-03-22T17:45:45Z | 2021-03-22T17:45:45Z | OWNER | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Use SQLite conn.interrupt() instead of sqlite_timelimit() 837956424 | ||
804263434 | https://github.com/simonw/datasette/issues/1249#issuecomment-804263434 | https://api.github.com/repos/simonw/datasette/issues/1249 | MDEyOklzc3VlQ29tbWVudDgwNDI2MzQzNA== | simonw 9599 | 2021-03-22T17:43:25Z | 2021-03-22T17:43:25Z | OWNER | I figured out the cause of the hang in #1268 - it was caused by |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Updated Dockerfile with SpatiaLite version 5.0 824064069 | |
804261915 | https://github.com/simonw/datasette/issues/1268#issuecomment-804261915 | https://api.github.com/repos/simonw/datasette/issues/1268 | MDEyOklzc3VlQ29tbWVudDgwNDI2MTkxNQ== | simonw 9599 | 2021-03-22T17:41:12Z | 2021-03-22T17:41:12Z | OWNER | Closing this because I've figured out the root of the problem now, and I have a potential solution. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Figure out why SpatiaLite 5.0 hangs the database page on Linux 837308703 | |
804261610 | https://github.com/simonw/datasette/issues/1269#issuecomment-804261610 | https://api.github.com/repos/simonw/datasette/issues/1269 | MDEyOklzc3VlQ29tbWVudDgwNDI2MTYxMA== | simonw 9599 | 2021-03-22T17:40:41Z | 2021-03-22T17:40:41Z | OWNER | 1270 looks promising, and I don't want to leave open a security hole where someone could potentially hang Datasette with a nasty |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Don't attempt to run count(*) against virtual tables 837348479 | |
804261103 | https://github.com/simonw/datasette/pull/1271#issuecomment-804261103 | https://api.github.com/repos/simonw/datasette/issues/1271 | MDEyOklzc3VlQ29tbWVudDgwNDI2MTEwMw== | codecov[bot] 22429695 | 2021-03-22T17:39:57Z | 2021-03-22T17:39:57Z | NONE | Codecov Report
```diff @@ Coverage Diff @@ main #1271 +/-==========================================
- Coverage 91.51% 91.22% -0.29% | Impacted Files | Coverage Δ | |
|---|---|---|
| datasette/database.py | Continue to review full report at Codecov.
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Use SQLite conn.interrupt() instead of sqlite_timelimit() 837956424 | |
804255633 | https://github.com/simonw/datasette/issues/1270#issuecomment-804255633 | https://api.github.com/repos/simonw/datasette/issues/1270 | MDEyOklzc3VlQ29tbWVudDgwNDI1NTYzMw== | simonw 9599 | 2021-03-22T17:32:02Z | 2021-03-22T17:32:08Z | OWNER | Confirmed that the |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Try implementing SQLite timeouts using .interrupt() instead of using .set_progress_handler() 837350092 | |
803834784 | https://github.com/simonw/datasette/issues/1270#issuecomment-803834784 | https://api.github.com/repos/simonw/datasette/issues/1270 | MDEyOklzc3VlQ29tbWVudDgwMzgzNDc4NA== | simonw 9599 | 2021-03-22T07:31:57Z | 2021-03-22T16:22:19Z | OWNER | I think the implementation for this goes here: https://github.com/simonw/datasette/blob/6f41c8a2bef309a66588b2875c3e24d26adb4850/datasette/database.py#L146-L157 I figured out a similar pattern in |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Try implementing SQLite timeouts using .interrupt() instead of using .set_progress_handler() 837350092 | |
803802957 | https://github.com/simonw/datasette/issues/1268#issuecomment-803802957 | https://api.github.com/repos/simonw/datasette/issues/1268 | MDEyOklzc3VlQ29tbWVudDgwMzgwMjk1Nw== | simonw 9599 | 2021-03-22T06:38:14Z | 2021-03-22T06:38:14Z | OWNER | Also worth trying is to change this code:
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Figure out why SpatiaLite 5.0 hangs the database page on Linux 837308703 | |
803785808 | https://github.com/simonw/datasette/issues/1269#issuecomment-803785808 | https://api.github.com/repos/simonw/datasette/issues/1269 | MDEyOklzc3VlQ29tbWVudDgwMzc4NTgwOA== | simonw 9599 | 2021-03-22T06:00:53Z | 2021-03-22T06:00:53Z | OWNER | This may not be necessary if using `.interrupt() for SQLite timeouts in #1270 works. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Don't attempt to run count(*) against virtual tables 837348479 | |
803784902 | https://github.com/simonw/datasette/issues/1268#issuecomment-803784902 | https://api.github.com/repos/simonw/datasette/issues/1268 | MDEyOklzc3VlQ29tbWVudDgwMzc4NDkwMg== | simonw 9599 | 2021-03-22T05:59:06Z | 2021-03-22T05:59:06Z | OWNER | Even if I implement that workaround in #1269 I'm concerned that this could still allow users to deliberately crash Datasette (if it's running SpatiaLite 5.0) by executing That |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Figure out why SpatiaLite 5.0 hangs the database page on Linux 837308703 | |
803782705 | https://github.com/simonw/datasette/issues/1268#issuecomment-803782705 | https://api.github.com/repos/simonw/datasette/issues/1268 | MDEyOklzc3VlQ29tbWVudDgwMzc4MjcwNQ== | simonw 9599 | 2021-03-22T05:54:19Z | 2021-03-22T05:54:19Z | OWNER | Got two new TILs out of this: |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Figure out why SpatiaLite 5.0 hangs the database page on Linux 837308703 | |
803777724 | https://github.com/simonw/datasette/issues/1268#issuecomment-803777724 | https://api.github.com/repos/simonw/datasette/issues/1268 | MDEyOklzc3VlQ29tbWVudDgwMzc3NzcyNA== | simonw 9599 | 2021-03-22T05:42:50Z | 2021-03-22T05:43:23Z | OWNER | If I want to avoid counting virtual tables, I need to detect which tables are virtual tables. The safest way to do this is probably to pull the This would catch things like |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Figure out why SpatiaLite 5.0 hangs the database page on Linux 837308703 | |
803775121 | https://github.com/simonw/datasette/issues/1268#issuecomment-803775121 | https://api.github.com/repos/simonw/datasette/issues/1268 | MDEyOklzc3VlQ29tbWVudDgwMzc3NTEyMQ== | simonw 9599 | 2021-03-22T05:36:26Z | 2021-03-22T05:36:26Z | OWNER | So one fix could be to avoid running counts for anything that turns out to be a virtual table. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Figure out why SpatiaLite 5.0 hangs the database page on Linux 837308703 | |
803774926 | https://github.com/simonw/datasette/issues/1268#issuecomment-803774926 | https://api.github.com/repos/simonw/datasette/issues/1268 | MDEyOklzc3VlQ29tbWVudDgwMzc3NDkyNg== | simonw 9599 | 2021-03-22T05:35:56Z | 2021-03-22T05:35:56Z | OWNER | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Figure out why SpatiaLite 5.0 hangs the database page on Linux 837308703 | ||
803774518 | https://github.com/simonw/datasette/issues/1268#issuecomment-803774518 | https://api.github.com/repos/simonw/datasette/issues/1268 | MDEyOklzc3VlQ29tbWVudDgwMzc3NDUxOA== | simonw 9599 | 2021-03-22T05:34:57Z | 2021-03-22T05:34:57Z | OWNER | ... and sure enough, adding this code fixed the problem:
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Figure out why SpatiaLite 5.0 hangs the database page on Linux 837308703 | |
803773484 | https://github.com/simonw/datasette/issues/1268#issuecomment-803773484 | https://api.github.com/repos/simonw/datasette/issues/1268 | MDEyOklzc3VlQ29tbWVudDgwMzc3MzQ4NA== | simonw 9599 | 2021-03-22T05:32:29Z | 2021-03-22T05:32:29Z | OWNER | To figure out which SQL query triggers the problem I added this code to write to a log file:
With the log enabled, I used
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Figure out why SpatiaLite 5.0 hangs the database page on Linux 837308703 | |
803764919 | https://github.com/simonw/datasette/issues/1268#issuecomment-803764919 | https://api.github.com/repos/simonw/datasette/issues/1268 | MDEyOklzc3VlQ29tbWVudDgwMzc2NDkxOQ== | simonw 9599 | 2021-03-22T05:11:11Z | 2021-03-22T05:11:11Z | OWNER | Maybe I could implement SQLite query timeouts using the https://stackoverflow.com/questions/43240496/python-sqlite3-how-to-quickly-and-cleanly-interrupt-long-running-query-with-e has some tips. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Figure out why SpatiaLite 5.0 hangs the database page on Linux 837308703 | |
803764200 | https://github.com/simonw/datasette/issues/1268#issuecomment-803764200 | https://api.github.com/repos/simonw/datasette/issues/1268 | MDEyOklzc3VlQ29tbWVudDgwMzc2NDIwMA== | simonw 9599 | 2021-03-22T05:09:13Z | 2021-03-22T05:09:13Z | OWNER | I tried building a container where the |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Figure out why SpatiaLite 5.0 hangs the database page on Linux 837308703 | |
803762969 | https://github.com/simonw/datasette/issues/1268#issuecomment-803762969 | https://api.github.com/repos/simonw/datasette/issues/1268 | MDEyOklzc3VlQ29tbWVudDgwMzc2Mjk2OQ== | simonw 9599 | 2021-03-22T05:05:51Z | 2021-03-22T05:05:51Z | OWNER | I had to run |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Figure out why SpatiaLite 5.0 hangs the database page on Linux 837308703 | |
803762609 | https://github.com/simonw/datasette/issues/1268#issuecomment-803762609 | https://api.github.com/repos/simonw/datasette/issues/1268 | MDEyOklzc3VlQ29tbWVudDgwMzc2MjYwOQ== | simonw 9599 | 2021-03-22T05:05:00Z | 2021-03-22T05:05:00Z | OWNER | Using https://til.simonwillison.net/docker/attach-bash-to-running-container - I figured out how to run apt-get install gdb python3-dbggdb /usr/bin/python3 -p 20
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Figure out why SpatiaLite 5.0 hangs the database page on Linux 837308703 | |
803759051 | https://github.com/simonw/datasette/issues/1268#issuecomment-803759051 | https://api.github.com/repos/simonw/datasette/issues/1268 | MDEyOklzc3VlQ29tbWVudDgwMzc1OTA1MQ== | simonw 9599 | 2021-03-22T04:55:22Z | 2021-03-22T04:55:22Z | OWNER | So I think there's a bug in the way the |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Figure out why SpatiaLite 5.0 hangs the database page on Linux 837308703 | |
803758793 | https://github.com/simonw/datasette/issues/1268#issuecomment-803758793 | https://api.github.com/repos/simonw/datasette/issues/1268 | MDEyOklzc3VlQ29tbWVudDgwMzc1ODc5Mw== | simonw 9599 | 2021-03-22T04:54:32Z | 2021-03-22T04:54:32Z | OWNER | Hitting http://localhost:8001/tuscany_housenumbers triggers the bug. It gets stuck in a loop that looks like this: Which looks to me like this code: https://github.com/simonw/datasette/blob/8e18c7943181f228ce5ebcea48deb59ce50bee1f/datasette/utils/init.py#L139-L158 |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Figure out why SpatiaLite 5.0 hangs the database page on Linux 837308703 | |
803758182 | https://github.com/simonw/datasette/issues/1268#issuecomment-803758182 | https://api.github.com/repos/simonw/datasette/issues/1268 | MDEyOklzc3VlQ29tbWVudDgwMzc1ODE4Mg== | simonw 9599 | 2021-03-22T04:52:15Z | 2021-03-22T04:52:15Z | OWNER | Hitting http://localhost:8001/ successfully shows the homepage (after a lot more scrolling). |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Figure out why SpatiaLite 5.0 hangs the database page on Linux 837308703 | |
803757746 | https://github.com/simonw/datasette/issues/1268#issuecomment-803757746 | https://api.github.com/repos/simonw/datasette/issues/1268 | MDEyOklzc3VlQ29tbWVudDgwMzc1Nzc0Ng== | simonw 9599 | 2021-03-22T04:50:40Z | 2021-03-22T04:51:52Z | OWNER | Here's a fun debugging trick:
A huge amount of stuff scrolls past as Datasette starts up, since we are tracing every executed line of Python. After about a minute it's finished starting and gets to this point:
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Figure out why SpatiaLite 5.0 hangs the database page on Linux 837308703 | |
803756495 | https://github.com/simonw/datasette/issues/1268#issuecomment-803756495 | https://api.github.com/repos/simonw/datasette/issues/1268 | MDEyOklzc3VlQ29tbWVudDgwMzc1NjQ5NQ== | simonw 9599 | 2021-03-22T04:46:04Z | 2021-03-22T04:46:04Z | OWNER |
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Figure out why SpatiaLite 5.0 hangs the database page on Linux 837308703 | |
803755698 | https://github.com/simonw/datasette/issues/1249#issuecomment-803755698 | https://api.github.com/repos/simonw/datasette/issues/1249 | MDEyOklzc3VlQ29tbWVudDgwMzc1NTY5OA== | simonw 9599 | 2021-03-22T04:43:02Z | 2021-03-22T04:43:02Z | OWNER | I'll spin off a separate ticket to investigate the hang. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Updated Dockerfile with SpatiaLite version 5.0 824064069 | |
803754226 | https://github.com/simonw/datasette/issues/1267#issuecomment-803754226 | https://api.github.com/repos/simonw/datasette/issues/1267 | MDEyOklzc3VlQ29tbWVudDgwMzc1NDIyNg== | simonw 9599 | 2021-03-22T04:37:26Z | 2021-03-22T04:37:26Z | OWNER | Thanks for doing this - I've used alternativeto.net a bunch in the past, it's great to see Datasette listed there. This does raise some interesting philosophical questions: three years into the project I'm still not entirely sure what Datasette competes with! Could be SQLite desktop packages, could be visualization software like Tableau, could even be something like Airtable (given a few more plugins). It will be interesting to see how the alternativeto listing evolves, maybe it will help me answer that question! |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Update Datasette alternativeto listening with details 837208901 | |
803753388 | https://github.com/simonw/datasette/issues/1249#issuecomment-803753388 | https://api.github.com/repos/simonw/datasette/issues/1249 | MDEyOklzc3VlQ29tbWVudDgwMzc1MzM4OA== | simonw 9599 | 2021-03-22T04:34:20Z | 2021-03-22T04:35:10Z | OWNER | Well this is frustrating. I finally found a Dockerfile that worked and installed an Ubuntu pre-compiled SpatiaLite module that would load... ```dockerfile FROM ubuntu:20.10 as install_spatialite RUN apt update && \ apt install -y libsqlite3-mod-spatialite && \ apt clean && \ rm -rf /var/lib/{apt,dpkg,cache,log}/ FROM ubuntu:20.10 RUN apt update && \ apt install -y python3-pip && \ apt clean && \ rm -rf /var/lib/{apt,dpkg,cache,log}/ RUN pip install datasette Copy spatial extensionsCOPY --from=install_spatialite /usr/lib/x86_64-linux-gnu /usr/lib/x86_64-linux-gnu/ ENV LD_LIBRARY_PATH=/usr/local/lib EXPOSE 8001 CMD ["datasette"] ``` (Which produced a 550MB image) And when I ran Datasette I got that same error where the database listing page hangs!
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Updated Dockerfile with SpatiaLite version 5.0 824064069 | |
803751068 | https://github.com/simonw/datasette/issues/1249#issuecomment-803751068 | https://api.github.com/repos/simonw/datasette/issues/1249 | MDEyOklzc3VlQ29tbWVudDgwMzc1MTA2OA== | simonw 9599 | 2021-03-22T04:26:45Z | 2021-03-22T04:26:45Z | OWNER | Here's why:
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Updated Dockerfile with SpatiaLite version 5.0 824064069 | |
803750617 | https://github.com/simonw/datasette/issues/1249#issuecomment-803750617 | https://api.github.com/repos/simonw/datasette/issues/1249 | MDEyOklzc3VlQ29tbWVudDgwMzc1MDYxNw== | simonw 9599 | 2021-03-22T04:25:14Z | 2021-03-22T04:25:14Z | OWNER | Got this error attempting to run Datasette (with or without SpatiaLite):
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Updated Dockerfile with SpatiaLite version 5.0 824064069 | |
803750399 | https://github.com/simonw/datasette/issues/1249#issuecomment-803750399 | https://api.github.com/repos/simonw/datasette/issues/1249 | MDEyOklzc3VlQ29tbWVudDgwMzc1MDM5OQ== | simonw 9599 | 2021-03-22T04:24:25Z | 2021-03-22T04:24:25Z | OWNER | I'll try using RUN apt update && \ apt install -y libsqlite3-mod-spatialite && \ apt clean && \ rm -rf /var/lib/{apt,dpkg,cache,log}/ FROM ubuntu:20.10 as build RUN apt update && \ apt install -y python3-pip && \ apt clean && \ rm -rf /var/lib/{apt,dpkg,cache,log}/ RUN pip install datasette COPY . /datasetteRUN pip install /datasetteFROM ubuntu:20.10 Copy python dependencies and spatialite librariesCOPY --from=build /usr/local/lib/ /usr/local/lib/ Copy executablesCOPY --from=build /usr/local/bin /usr/local/bin Copy spatial extensionsCOPY --from=install_spatialite /usr/lib/x86_64-linux-gnu/mod_spatialite.so /usr/lib/x86_64-linux-gnu/mod_spatialite.so ENV LD_LIBRARY_PATH=/usr/local/lib EXPOSE 8001 CMD ["datasette"] ``` |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Updated Dockerfile with SpatiaLite version 5.0 824064069 | |
803749831 | https://github.com/simonw/datasette/issues/1249#issuecomment-803749831 | https://api.github.com/repos/simonw/datasette/issues/1249 | MDEyOklzc3VlQ29tbWVudDgwMzc0OTgzMQ== | simonw 9599 | 2021-03-22T04:22:35Z | 2021-03-22T04:22:35Z | OWNER | I tried copying just the RUN apt update && \ apt install -y libsqlite3-mod-spatialite && \ apt clean && \ rm -rf /var/lib/{apt,dpkg,cache,log}/ FROM python:3.9.2-slim as build RUN pip install datasette COPY . /datasetteRUN pip install /datasetteFROM python:3.9.2-slim Copy python dependencies and spatialite librariesCOPY --from=build /usr/local/lib/ /usr/local/lib/ Copy executablesCOPY --from=build /usr/local/bin /usr/local/bin Copy spatial extensionsCOPY --from=install_spatialite /usr/lib/x86_64-linux-gnu/mod_spatialite.so /usr/lib/x86_64-linux-gnu/mod_spatialite.so ENV LD_LIBRARY_PATH=/usr/local/lib EXPOSE 8001
CMD ["datasette"]
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Updated Dockerfile with SpatiaLite version 5.0 824064069 | |
803748469 | https://github.com/simonw/datasette/issues/1249#issuecomment-803748469 | https://api.github.com/repos/simonw/datasette/issues/1249 | MDEyOklzc3VlQ29tbWVudDgwMzc0ODQ2OQ== | simonw 9599 | 2021-03-22T04:17:51Z | 2021-03-22T04:17:51Z | OWNER | ... except my clever image using SpatiaLite installed for Ubuntu doesn't actually work:
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Updated Dockerfile with SpatiaLite version 5.0 824064069 | |
803748158 | https://github.com/simonw/datasette/issues/1249#issuecomment-803748158 | https://api.github.com/repos/simonw/datasette/issues/1249 | MDEyOklzc3VlQ29tbWVudDgwMzc0ODE1OA== | simonw 9599 | 2021-03-22T04:16:57Z | 2021-03-22T04:16:57Z | OWNER | Which is great, because the image on Docker Hub right now is 383MB. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Updated Dockerfile with SpatiaLite version 5.0 824064069 | |
803747701 | https://github.com/simonw/datasette/issues/1249#issuecomment-803747701 | https://api.github.com/repos/simonw/datasette/issues/1249 | MDEyOklzc3VlQ29tbWVudDgwMzc0NzcwMQ== | simonw 9599 | 2021-03-22T04:15:40Z | 2021-03-22T04:15:40Z | OWNER | Here's a trick: install SpatiaLite in ```dockerfile FROM ubuntu:20.10 as install_spatialite RUN apt update && \ apt install -y libsqlite3-mod-spatialite && \ apt clean && \ rm -rf /var/lib/{apt,dpkg,cache,log}/ FROM python:3.9.2-slim as build RUN pip install datasette COPY . /datasetteRUN pip install /datasetteFROM python:3.9.2-slim Copy python dependencies and spatialite librariesCOPY --from=build /usr/local/lib/ /usr/local/lib/ Copy executablesCOPY --from=build /usr/local/bin /usr/local/bin Copy spatial extensionsCOPY --from=install_spatialite /usr/lib/x86_64-linux-gnu /usr/lib/x86_64-linux-gnu ENV LD_LIBRARY_PATH=/usr/local/lib EXPOSE 8001 CMD ["datasette"] ``` That produced a 265MB image. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Updated Dockerfile with SpatiaLite version 5.0 824064069 | |
803700940 | https://github.com/simonw/datasette/issues/1249#issuecomment-803700940 | https://api.github.com/repos/simonw/datasette/issues/1249 | MDEyOklzc3VlQ29tbWVudDgwMzcwMDk0MA== | simonw 9599 | 2021-03-22T01:14:24Z | 2021-03-22T01:14:24Z | OWNER | I tried that with just |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Updated Dockerfile with SpatiaLite version 5.0 824064069 | |
803700626 | https://github.com/simonw/datasette/issues/1249#issuecomment-803700626 | https://api.github.com/repos/simonw/datasette/issues/1249 | MDEyOklzc3VlQ29tbWVudDgwMzcwMDYyNg== | simonw 9599 | 2021-03-22T01:13:04Z | 2021-03-22T01:13:04Z | OWNER | Building a Dockerfile containing just Building this one: ```dockerfile FROM ubuntu:20.10 Setup build dependenciesRUN apt update && \ apt install -y python3-pip libsqlite3-mod-spatialite && \ apt clean && \ rm -rf /var/lib/{apt,dpkg,cache,log}/ ``` Resulted in a 515MB image. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Updated Dockerfile with SpatiaLite version 5.0 824064069 | |
803698983 | https://github.com/simonw/datasette/issues/1249#issuecomment-803698983 | https://api.github.com/repos/simonw/datasette/issues/1249 | MDEyOklzc3VlQ29tbWVudDgwMzY5ODk4Mw== | simonw 9599 | 2021-03-22T01:05:36Z | 2021-03-22T01:06:23Z | OWNER | It's pretty big though. I tried this version which avoids copying junk from my laptop in: ```dockerfile FROM ubuntu:20.10 Setup build dependenciesRUN apt update && apt install -y python3-pip libsqlite3-mod-spatialite && apt clean RUN pip install datasette EXPOSE 8001
CMD ["datasette"]
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Updated Dockerfile with SpatiaLite version 5.0 824064069 | |
803698168 | https://github.com/simonw/datasette/issues/1249#issuecomment-803698168 | https://api.github.com/repos/simonw/datasette/issues/1249 | MDEyOklzc3VlQ29tbWVudDgwMzY5ODE2OA== | simonw 9599 | 2021-03-22T01:02:02Z | 2021-03-22T01:02:30Z | OWNER | This is the shortest Dockerfile that appeared to give me a working SpatiaLite module: ```dockerfile FROM ubuntu:20.10 Setup build dependenciesRUN apt update && apt install -y python3-pip libsqlite3-mod-spatialite && apt clean Add local code to the image instead of fetching from pypi.COPY . /datasette RUN pip install /datasette RUN rm -rf /datasette EXPOSE 8001 CMD ["datasette"] ``` |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Updated Dockerfile with SpatiaLite version 5.0 824064069 | |
803697546 | https://github.com/simonw/datasette/issues/1249#issuecomment-803697546 | https://api.github.com/repos/simonw/datasette/issues/1249 | MDEyOklzc3VlQ29tbWVudDgwMzY5NzU0Ng== | simonw 9599 | 2021-03-22T00:59:47Z | 2021-03-22T00:59:47Z | OWNER | To debug I'm running:
This gets me a shell I can use. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Updated Dockerfile with SpatiaLite version 5.0 824064069 | |
803697211 | https://github.com/simonw/datasette/issues/1249#issuecomment-803697211 | https://api.github.com/repos/simonw/datasette/issues/1249 | MDEyOklzc3VlQ29tbWVudDgwMzY5NzIxMQ== | simonw 9599 | 2021-03-22T00:58:01Z | 2021-03-22T00:58:01Z | OWNER | I'm messing around with the
And then:
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Updated Dockerfile with SpatiaLite version 5.0 824064069 | |
803694661 | https://github.com/simonw/datasette/issues/1249#issuecomment-803694661 | https://api.github.com/repos/simonw/datasette/issues/1249 | MDEyOklzc3VlQ29tbWVudDgwMzY5NDY2MQ== | simonw 9599 | 2021-03-22T00:46:49Z | 2021-03-22T00:46:49Z | OWNER | Actually for the loadable module I think I need https://packages.ubuntu.com/groovy/libsqlite3-mod-spatialite |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Updated Dockerfile with SpatiaLite version 5.0 824064069 | |
803694436 | https://github.com/simonw/datasette/issues/1249#issuecomment-803694436 | https://api.github.com/repos/simonw/datasette/issues/1249 | MDEyOklzc3VlQ29tbWVudDgwMzY5NDQzNg== | simonw 9599 | 2021-03-22T00:46:00Z | 2021-03-22T00:46:00Z | OWNER | So I'm going to try |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Updated Dockerfile with SpatiaLite version 5.0 824064069 | |
803694359 | https://github.com/simonw/datasette/issues/1249#issuecomment-803694359 | https://api.github.com/repos/simonw/datasette/issues/1249 | MDEyOklzc3VlQ29tbWVudDgwMzY5NDM1OQ== | simonw 9599 | 2021-03-22T00:45:47Z | 2021-03-22T00:45:47Z | OWNER | https://pythonspeed.com/articles/base-image-python-docker-images/ suggests using |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Updated Dockerfile with SpatiaLite version 5.0 824064069 | |
803693181 | https://github.com/simonw/datasette/issues/1249#issuecomment-803693181 | https://api.github.com/repos/simonw/datasette/issues/1249 | MDEyOklzc3VlQ29tbWVudDgwMzY5MzE4MQ== | simonw 9599 | 2021-03-22T00:41:02Z | 2021-03-22T00:41:02Z | OWNER | Debian sid has it too: https://packages.debian.org/sid/libspatialite-dev |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Updated Dockerfile with SpatiaLite version 5.0 824064069 | |
803692673 | https://github.com/simonw/datasette/issues/1249#issuecomment-803692673 | https://api.github.com/repos/simonw/datasette/issues/1249 | MDEyOklzc3VlQ29tbWVudDgwMzY5MjY3Mw== | simonw 9599 | 2021-03-22T00:38:42Z | 2021-03-22T00:38:42Z | OWNER | Ubuntu Groovy has a package for SpatiaLite 5 - I could try using that instead: https://packages.ubuntu.com/groovy/libspatialite-dev |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Updated Dockerfile with SpatiaLite version 5.0 824064069 | |
803691236 | https://github.com/simonw/datasette/issues/1249#issuecomment-803691236 | https://api.github.com/repos/simonw/datasette/issues/1249 | MDEyOklzc3VlQ29tbWVudDgwMzY5MTIzNg== | simonw 9599 | 2021-03-22T00:32:03Z | 2021-03-22T00:32:03Z | OWNER | Here's something odd: when I run But when I tried compiling SpatiaLite inside the Docker container I had hanging errors on some pages. This is using https://www.gaia-gis.it/gaia-sins/knn/tuscany_housenumbers.7z from the SpatiaLite KNN tutorial at https://www.gaia-gis.it/fossil/libspatialite/wiki?name=KNN |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Updated Dockerfile with SpatiaLite version 5.0 824064069 | |
803674728 | https://github.com/simonw/datasette/issues/1259#issuecomment-803674728 | https://api.github.com/repos/simonw/datasette/issues/1259 | MDEyOklzc3VlQ29tbWVudDgwMzY3NDcyOA== | simonw 9599 | 2021-03-21T22:55:31Z | 2021-03-21T22:55:31Z | OWNER | CTEs were added in 2014-02-03 SQLite 3.8.3 - so I think it's OK to depend on them for Datasette. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Research using CTEs for faster facet counts 830567275 | |
803673225 | https://github.com/simonw/datasette/issues/647#issuecomment-803673225 | https://api.github.com/repos/simonw/datasette/issues/647 | MDEyOklzc3VlQ29tbWVudDgwMzY3MzIyNQ== | simonw 9599 | 2021-03-21T22:44:19Z | 2021-03-21T22:44:19Z | OWNER | Now that I'm looking at refactoring how views work in #878 it's clear that the gnarliest, most convoluted code I need to deal with relates to this old feature. I'm going to remove it entirely. Any performance enhancement or provides can be achieved just as well by using regular URLs and a caching proxy. I may provide a 404 handling plugin that attempts to rewrite old URLs that used this mechanism, but I won't do any more than that. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Move hashed URL mode out to a plugin 531755959 | |
803631102 | https://github.com/simonw/datasette/issues/942#issuecomment-803631102 | https://api.github.com/repos/simonw/datasette/issues/942 | MDEyOklzc3VlQ29tbWVudDgwMzYzMTEwMg== | mroswell 192568 | 2021-03-21T17:48:42Z | 2021-03-21T17:48:42Z | CONTRIBUTOR | I like this idea. Though it might be nice to have some kind of automated system from database to file, so that developers could easily track diffs. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Support column descriptions in metadata.json 681334912 | |
803502424 | https://github.com/simonw/sqlite-utils/issues/249#issuecomment-803502424 | https://api.github.com/repos/simonw/sqlite-utils/issues/249 | MDEyOklzc3VlQ29tbWVudDgwMzUwMjQyNA== | prabhur 36287 | 2021-03-21T02:43:32Z | 2021-03-21T02:43:32Z | NONE |
Wow. Wasn't expecting a response this quick, especially during a weekend. :-) Sincerely appreciate it.
I tried the |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Full text search possibly broken? 836963850 | |
803501756 | https://github.com/simonw/sqlite-utils/issues/249#issuecomment-803501756 | https://api.github.com/repos/simonw/sqlite-utils/issues/249 | MDEyOklzc3VlQ29tbWVudDgwMzUwMTc1Ng== | simonw 9599 | 2021-03-21T02:33:45Z | 2021-03-21T02:33:45Z | OWNER | Did you run If so you'll need to run A better solution may be to add |
{ "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Full text search possibly broken? 836963850 | |
803499509 | https://github.com/simonw/datasette/issues/1261#issuecomment-803499509 | https://api.github.com/repos/simonw/datasette/issues/1261 | MDEyOklzc3VlQ29tbWVudDgwMzQ5OTUwOQ== | brimstone 812795 | 2021-03-21T02:06:43Z | 2021-03-21T02:06:43Z | NONE | I can confirm 0.9.2 fixes the problem. Thanks for the fast response! |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Some links aren't properly URL encoded. 832092321 | |
803473015 | https://github.com/simonw/datasette/issues/878#issuecomment-803473015 | https://api.github.com/repos/simonw/datasette/issues/878 | MDEyOklzc3VlQ29tbWVudDgwMzQ3MzAxNQ== | simonw 9599 | 2021-03-20T22:33:05Z | 2021-03-20T22:33:05Z | OWNER | Things this mechanism needs to be able to support:
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
New pattern for views that return either JSON or HTML, available for plugins 648435885 | |
803472595 | https://github.com/simonw/datasette/issues/878#issuecomment-803472595 | https://api.github.com/repos/simonw/datasette/issues/878 | MDEyOklzc3VlQ29tbWVudDgwMzQ3MjU5NQ== | simonw 9599 | 2021-03-20T22:28:12Z | 2021-03-20T22:28:12Z | OWNER | Another idea I had: a view is a class that takes the |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
New pattern for views that return either JSON or HTML, available for plugins 648435885 | |
803472278 | https://github.com/simonw/datasette/issues/878#issuecomment-803472278 | https://api.github.com/repos/simonw/datasette/issues/878 | MDEyOklzc3VlQ29tbWVudDgwMzQ3MjI3OA== | simonw 9599 | 2021-03-20T22:25:04Z | 2021-03-20T22:25:04Z | OWNER | I came up with a slightly wild idea for this that would involve pytest-style dependency injection. Prototype here: https://gist.github.com/simonw/496b24fdad44f6f8b7237fe394a0ced7 Copying from my private notes:
|
{ "total_count": 1, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 1 } |
New pattern for views that return either JSON or HTML, available for plugins 648435885 | |
803471917 | https://github.com/simonw/datasette/issues/878#issuecomment-803471917 | https://api.github.com/repos/simonw/datasette/issues/878 | MDEyOklzc3VlQ29tbWVudDgwMzQ3MTkxNw== | simonw 9599 | 2021-03-20T22:21:33Z | 2021-03-20T22:21:33Z | OWNER | This has been blocking things for too long. If this becomes a documented pattern, things like adding a JSON output to https://github.com/dogsheep/dogsheep-beta becomes easier too. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
New pattern for views that return either JSON or HTML, available for plugins 648435885 | |
803471702 | https://github.com/simonw/datasette/issues/1258#issuecomment-803471702 | https://api.github.com/repos/simonw/datasette/issues/1258 | MDEyOklzc3VlQ29tbWVudDgwMzQ3MTcwMg== | simonw 9599 | 2021-03-20T22:19:39Z | 2021-03-20T22:19:39Z | OWNER | This is a good idea. I avoided this initially because it should be possible to run a canned query with a parameter set to the empty string, but that view could definitely be smart enough to differentiate between |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Allow canned query params to specify default values 828858421 | |
803469623 | https://github.com/simonw/datasette/issues/782#issuecomment-803469623 | https://api.github.com/repos/simonw/datasette/issues/782 | MDEyOklzc3VlQ29tbWVudDgwMzQ2OTYyMw== | simonw 9599 | 2021-03-20T22:01:23Z | 2021-03-20T22:01:23Z | OWNER | I'm going to keep |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Redesign default .json format 627794879 | |
803468314 | https://github.com/simonw/datasette/issues/1261#issuecomment-803468314 | https://api.github.com/repos/simonw/datasette/issues/1261 | MDEyOklzc3VlQ29tbWVudDgwMzQ2ODMxNA== | simonw 9599 | 2021-03-20T21:48:48Z | 2021-03-20T21:48:48Z | OWNER | That's fixed in this release of |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Some links aren't properly URL encoded. 832092321 | |
803466868 | https://github.com/simonw/datasette/issues/1261#issuecomment-803466868 | https://api.github.com/repos/simonw/datasette/issues/1261 | MDEyOklzc3VlQ29tbWVudDgwMzQ2Njg2OA== | simonw 9599 | 2021-03-20T21:36:06Z | 2021-03-20T21:36:06Z | OWNER | This isn't a Datasette bug - it's a Vercel bug: https://github.com/simonw/datasette-publish-vercel/issues/28 I'm looking at a fix for that now, so watch that issue for updates. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Some links aren't properly URL encoded. 832092321 | |
803466730 | https://github.com/simonw/datasette/issues/1266#issuecomment-803466730 | https://api.github.com/repos/simonw/datasette/issues/1266 | MDEyOklzc3VlQ29tbWVudDgwMzQ2NjczMA== | simonw 9599 | 2021-03-20T21:35:00Z | 2021-03-20T21:35:00Z | OWNER | { "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Documentation for Response.asgi_send(send) method 836273891 | ||
803160804 | https://github.com/simonw/datasette/issues/1265#issuecomment-803160804 | https://api.github.com/repos/simonw/datasette/issues/1265 | MDEyOklzc3VlQ29tbWVudDgwMzE2MDgwNA== | yunzheng 468612 | 2021-03-19T22:05:12Z | 2021-03-19T22:05:12Z | NONE | Wow that was fast! Thanks for this very cool project and quick update! 👍 |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Support for HTTP Basic Authentication 836123030 | |
803130332 | https://github.com/simonw/datasette/issues/1265#issuecomment-803130332 | https://api.github.com/repos/simonw/datasette/issues/1265 | MDEyOklzc3VlQ29tbWVudDgwMzEzMDMzMg== | simonw 9599 | 2021-03-19T21:03:09Z | 2021-03-19T21:03:09Z | OWNER | This is now available in |
{ "total_count": 1, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 1, "eyes": 0 } |
Support for HTTP Basic Authentication 836123030 | |
802923254 | https://github.com/simonw/datasette/issues/1265#issuecomment-802923254 | https://api.github.com/repos/simonw/datasette/issues/1265 | MDEyOklzc3VlQ29tbWVudDgwMjkyMzI1NA== | bobwhitelock 7476523 | 2021-03-19T15:39:15Z | 2021-03-19T15:39:15Z | CONTRIBUTOR | It doesn't use basic auth, but you can put a whole datasette instance, or parts of this, behind a username/password prompt using https://github.com/simonw/datasette-auth-passwords |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Support for HTTP Basic Authentication 836123030 | |
802164134 | https://github.com/simonw/datasette/issues/1262#issuecomment-802164134 | https://api.github.com/repos/simonw/datasette/issues/1262 | MDEyOklzc3VlQ29tbWVudDgwMjE2NDEzNA== | henry501 19328961 | 2021-03-18T17:55:00Z | 2021-03-18T17:55:00Z | NONE | Thanks for the comments. I'll take a look at the documentation to familiarize myself, as I haven't tried to write any plugins yet. With some luck I might be ready to write it when the hook is implemented. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Plugin hook that could support 'order by random()' for table view 834602299 | |
802099264 | https://github.com/simonw/datasette/issues/1262#issuecomment-802099264 | https://api.github.com/repos/simonw/datasette/issues/1262 | MDEyOklzc3VlQ29tbWVudDgwMjA5OTI2NA== | simonw 9599 | 2021-03-18T16:43:09Z | 2021-03-18T16:43:09Z | OWNER | I often find myself wanting this too, when I'm exploring a new dataset. i agree with Bob that this is a good candidate for a plugin. The plugin system isn't quite setup for this yet though - there isn't an obvious mechanism for adding extra sort orders or other interface elements that manipulate the query used by the table view in some way. I'm going to promote this issue to status of a plugin hook feature request - I have a hunch that a plugin hook that enables |
{ "total_count": 2, "+1": 2, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Plugin hook that could support 'order by random()' for table view 834602299 | |
802095132 | https://github.com/simonw/datasette/issues/1262#issuecomment-802095132 | https://api.github.com/repos/simonw/datasette/issues/1262 | MDEyOklzc3VlQ29tbWVudDgwMjA5NTEzMg== | bobwhitelock 7476523 | 2021-03-18T16:37:45Z | 2021-03-18T16:37:45Z | CONTRIBUTOR | This sounds like a good use case for a plugin, since this will only be useful for a subset of Datasette users. It shouldn't be too difficult to add a button to do this with the available plugin hooks - have you taken a look at https://docs.datasette.io/en/latest/writing_plugins.html? |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Plugin hook that could support 'order by random()' for table view 834602299 | |
802032152 | https://github.com/simonw/sqlite-utils/issues/159#issuecomment-802032152 | https://api.github.com/repos/simonw/sqlite-utils/issues/159 | MDEyOklzc3VlQ29tbWVudDgwMjAzMjE1Mg== | limar 1025224 | 2021-03-18T15:42:52Z | 2021-03-18T15:42:52Z | NONE | I confirm the bug. Happens for me in version 3.6. I use the call to delete all the records:
I see that |
{ "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
.delete_where() does not auto-commit (unlike .insert() or .upsert()) 702386948 | |
801816980 | https://github.com/simonw/sqlite-utils/issues/246#issuecomment-801816980 | https://api.github.com/repos/simonw/sqlite-utils/issues/246 | MDEyOklzc3VlQ29tbWVudDgwMTgxNjk4MA== | polyrand 37962604 | 2021-03-18T10:40:32Z | 2021-03-18T10:43:04Z | NONE | I have found a similar problem, but I only when using that type of query (with
I thought I could build a query like the first one using this function:
And then I use the output of that function as the query parameter for the standard However, my use case is different because I'm the one "deciding" when to use a prefix search, not the end user. I also haven't done many tests, but maybe you found that useful. One thing I could think of is checking if the query has an This is just for prefix queries, I think having the escaping function is still useful for other use cases. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Escaping FTS search strings 831751367 | |
799479175 | https://github.com/simonw/sqlite-utils/issues/246#issuecomment-799479175 | https://api.github.com/repos/simonw/sqlite-utils/issues/246 | MDEyOklzc3VlQ29tbWVudDc5OTQ3OTE3NQ== | simonw 9599 | 2021-03-15T14:47:31Z | 2021-03-15T14:47:31Z | OWNER | This is a smart feature. I have something that does this in Datasette, extracting it out to https://github.com/simonw/datasette/blob/8e18c7943181f228ce5ebcea48deb59ce50bee1f/datasette/utils/init.py#L818-L829 |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Escaping FTS search strings 831751367 | |
799066252 | https://github.com/simonw/datasette/issues/236#issuecomment-799066252 | https://api.github.com/repos/simonw/datasette/issues/236 | MDEyOklzc3VlQ29tbWVudDc5OTA2NjI1Mg== | simonw 9599 | 2021-03-15T03:34:52Z | 2021-03-15T03:34:52Z | OWNER | Yeah the Lambda Docker stuff is pretty odd - you still don't get to speak HTTP, you have to speak their custom event protocol instead. https://github.com/glassechidna/serverlessish looks interesting here - it adds a proxy inside the container which allows your existing HTTP Docker image to run within Docker-on-Lambda. I've not tried it out yet though. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
datasette publish lambda plugin 317001500 | |
799003172 | https://github.com/simonw/datasette/issues/236#issuecomment-799003172 | https://api.github.com/repos/simonw/datasette/issues/236 | MDEyOklzc3VlQ29tbWVudDc5OTAwMzE3Mg== | jacobian 21148 | 2021-03-14T23:42:57Z | 2021-03-14T23:42:57Z | CONTRIBUTOR | Oh, and the container image can be up to 10GB, so the EFS step might not be needed except for pretty big stuff. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
datasette publish lambda plugin 317001500 | |
799002993 | https://github.com/simonw/datasette/issues/236#issuecomment-799002993 | https://api.github.com/repos/simonw/datasette/issues/236 | MDEyOklzc3VlQ29tbWVudDc5OTAwMjk5Mw== | jacobian 21148 | 2021-03-14T23:41:51Z | 2021-03-14T23:41:51Z | CONTRIBUTOR | Now that Lambda supports Docker, this probably is a bit easier and may be able to build on top of the existing package command. There are weirdnesses in how the command actually gets invoked; the aws-lambda-python image shows a bit of that. So Datasette would probably need some sort of Lambda-specific entry point to make this work. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
datasette publish lambda plugin 317001500 | |
798913090 | https://github.com/simonw/datasette/pull/1260#issuecomment-798913090 | https://api.github.com/repos/simonw/datasette/issues/1260 | MDEyOklzc3VlQ29tbWVudDc5ODkxMzA5MA== | codecov[bot] 22429695 | 2021-03-14T14:01:30Z | 2021-03-14T14:01:30Z | NONE | Codecov Report
```diff @@ Coverage Diff @@ main #1260 +/-=======================================
Coverage 91.51% 91.51% | Impacted Files | Coverage Δ | |
|---|---|---|
| datasette/inspect.py | Continue to review full report at Codecov.
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Fix: code quality issues 831163537 | |
798468572 | https://github.com/dogsheep/healthkit-to-sqlite/issues/14#issuecomment-798468572 | https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/14 | MDEyOklzc3VlQ29tbWVudDc5ODQ2ODU3Mg== | n8henrie 1234956 | 2021-03-13T14:47:31Z | 2021-03-13T14:47:31Z | NONE | Ok, new PR works. I'm not I still end up with a lot of activities that are missing an
I also end up with some unhappy characters (in the skipped events), such as: But it's successfully making it through the file, and the resulting db opens in datasette, so I'd call that progress. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
UNIQUE constraint failed: workouts.id 771608692 | |
798436026 | https://github.com/dogsheep/healthkit-to-sqlite/issues/14#issuecomment-798436026 | https://api.github.com/repos/dogsheep/healthkit-to-sqlite/issues/14 | MDEyOklzc3VlQ29tbWVudDc5ODQzNjAyNg== | n8henrie 1234956 | 2021-03-13T14:23:16Z | 2021-03-13T14:23:16Z | NONE | This PR allows my import to succeed. It looks like some events don't have an If a record has neither of these, I changed it to just print the record (for debugging) and For some odd reason this ran fine at first, and now (after removing the generated db and trying again) I'm getting a different error (duplicate column name). Looks like it may have run when I had two successive runs without remembering to delete the db in between. Will try to refactor. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
UNIQUE constraint failed: workouts.id 771608692 | |
797827038 | https://github.com/simonw/datasette/issues/1259#issuecomment-797827038 | https://api.github.com/repos/simonw/datasette/issues/1259 | MDEyOklzc3VlQ29tbWVudDc5NzgyNzAzOA== | simonw 9599 | 2021-03-13T00:15:40Z | 2021-03-13T00:15:40Z | OWNER | If all of the facets were being calculated in a single query, I'd be willing to bump the facet time limit up to something a lot higher, maybe even a full second. There's a chance that could work amazingly well with a materialized CTE. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Research using CTEs for faster facet counts 830567275 | |
797804869 | https://github.com/simonw/datasette/issues/1259#issuecomment-797804869 | https://api.github.com/repos/simonw/datasette/issues/1259 | MDEyOklzc3VlQ29tbWVudDc5NzgwNDg2OQ== | simonw 9599 | 2021-03-12T23:05:05Z | 2021-03-12T23:05:05Z | OWNER | I wonder if I could optimize facet suggestion in the same way? One challenge: the query time limit will apply to the full CTE query, not to the individual columns. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Research using CTEs for faster facet counts 830567275 |
Advanced export
JSON shape: default, array, newline-delimited, object
CREATE TABLE [issue_comments] ( [html_url] TEXT, [issue_url] TEXT, [id] INTEGER PRIMARY KEY, [node_id] TEXT, [user] INTEGER REFERENCES [users]([id]), [created_at] TEXT, [updated_at] TEXT, [author_association] TEXT, [body] TEXT, [reactions] TEXT, [issue] INTEGER REFERENCES [issues]([id]) , [performed_via_github_app] TEXT); CREATE INDEX [idx_issue_comments_issue] ON [issue_comments] ([issue]); CREATE INDEX [idx_issue_comments_user] ON [issue_comments] ([user]);
user >30