This would be helpful.

Happy birthday to datasette and thank you Simon for your continued effort on this project!

I use datasette (python) as a fast layer on top of search for github projects using https://github.com/dogsheep/github-to-sqlite , and use the JSON API it provides to serve sample data to make Vega-Lite graphing workshop examples that don't require authentication/API keys. It's awesome to have a full SQL API support working without needing to develop any custom API middleware for both filtering and grouping.

I've also enjoyed using it as a teaching tool for working with public dataset in civic data workshops and as a platform for making visualization plugins . I

I'm especially excited about datasette-lite, as it will let people participate in future editions of this workshop without having to install anything to make use of their own tables :)

I guess it is not incorrect when it says the version is 4, though it is confusing. Maybe it doesn't even refer to FTS4/FTS5 versions, but something else? In any case, it's not related to sqlite-utils, but SQLite itself.

Okay, my final observations for the night! I've been pushing and pulling the various levers in utils/__init__.py to see what makes this work without hard-coding in something for arm64 and it seems that if I change /usr/lib/x86_64-linux-gnu/mod_spatialite.so here to just mod_spatialite it's happy.

Unfortunately cannot audit that for x86_64, but maybe that's a solution that'd be cross-arch compatible? It seems like it's the hard-coding of that path that's tripping it up.

(It was actually this comment from back in 2018 in an entirely unrelated repo that nudged me to try this, ha.)

Can confirm that my uname -a returns something different at the end:

root:xnu-8792.41.9~2/RELEASE_ARM64_T6000 arm64

I'm in arm64 land, you're in x86_64. I am admittedly very fuzzy on how this factors into Docker these days. Honestly thought this was one of the things Docker was suppose to help address. 🤔

Is it, uh, possible we are on different architectures? 😅 I'm using an Apple M1 Pro.

I jumped into a bash shell of an unmodified python:3.11.0-slim-bullseye container and manually ran apt-get update and installed libsqlite3-mod-spatialite. I don't end up with with mod_spatialite.so in /usr/lib/x86_64-linux-gnu/ — mine is in /usr/lib/aarch64-linux-gnu/.

I swapped that directory in here in a local copy of datasette and poof — it worked!

Interesting! So I tried this locally using your copy of nps-spatialite.db and I got the same error. 🤔

``` ❯ datasette package nps-spatialite.db --spatialite [+] Building 27.5s (10/10) FINISHED
=> [internal] load build definition from Dockerfile 0.0s => => transferring dockerfile: 622B 0.0s => [internal] load .dockerignore 0.0s => => transferring context: 2B 0.0s => [internal] load metadata for docker.io/library/python:3.11.0-slim-bullseye 0.9s => [internal] load build context 2.3s => => transferring context: 72.38MB 2.3s => CACHED [1/6] FROM docker.io/library/python:3.11.0-slim-bullseye@sha256:1cd45c5dad845af18d71745c017325725dc979571c1bbe625b67e6051533716c 0.0s => [2/6] COPY . /app 0.1s => [3/6] WORKDIR /app 0.0s => [4/6] RUN apt-get update && apt-get install -y python3-dev gcc libsqlite3-mod-spatialite && rm -rf /var/lib/apt/lists/* 18.5s => [5/6] RUN pip install -U datasette 4.9s => ERROR [6/6] RUN datasette inspect nps-spatialite.db --inspect-file inspect-data.json 0.7s

[6/6] RUN datasette inspect nps-spatialite.db --inspect-file inspect-data.json:

10 0.681 Traceback (most recent call last):

10 0.681 File "/usr/local/bin/datasette", line 8, in <module>

10 0.681 sys.exit(cli())

10 0.681 ^^^^^

10 0.681 File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1130, in call

10 0.682 return self.main(args, *kwargs)

10 0.682 ^^^^^^^^^^^^^^^^^^^^^^^^^^

10 0.682 File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1055, in main

10 0.682 rv = self.invoke(ctx)

10 0.682 ^^^^^^^^^^^^^^^^

10 0.682 File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1657, in invoke

10 0.682 return _process_result(sub_ctx.command.invoke(sub_ctx))

10 0.682 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

10 0.682 File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1404, in invoke

10 0.682 return ctx.invoke(self.callback, **ctx.params)

10 0.682 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

10 0.682 File "/usr/local/lib/python3.11/site-packages/click/core.py", line 760, in invoke

10 0.682 return __callback(args, *kwargs)

10 0.682 ^^^^^^^^^^^^^^^^^^^^^^^^^^^

10 0.683 File "/usr/local/lib/python3.11/site-packages/datasette/cli.py", line 164, in inspect

10 0.683 inspect_data = loop.run_until_complete(inspect_(files, sqlite_extensions))

10 0.683 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

10 0.683 File "/usr/local/lib/python3.11/asyncio/base_events.py", line 650, in run_until_complete

10 0.683 return future.result()

10 0.683 ^^^^^^^^^^^^^^^

10 0.683 File "/usr/local/lib/python3.11/site-packages/datasette/cli.py", line 179, in inspect_

10 0.683 counts = await database.table_counts(limit=3600 * 1000)

10 0.683 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

10 0.683 File "/usr/local/lib/python3.11/site-packages/datasette/database.py", line 304, in table_counts

10 0.683 for table in await self.table_names():

10 0.683 ^^^^^^^^^^^^^^^^^^^^^^^^

10 0.683 File "/usr/local/lib/python3.11/site-packages/datasette/database.py", line 342, in table_names

10 0.683 results = await self.execute(

10 0.683 ^^^^^^^^^^^^^^^^^^^

10 0.683 File "/usr/local/lib/python3.11/site-packages/datasette/database.py", line 267, in execute

10 0.683 results = await self.execute_fn(sql_operation_in_thread)

10 0.683 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

10 0.683 File "/usr/local/lib/python3.11/site-packages/datasette/database.py", line 213, in execute_fn

10 0.683 return await asyncio.get_event_loop().run_in_executor(

10 0.683 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

10 0.683 File "/usr/local/lib/python3.11/concurrent/futures/thread.py", line 58, in run

10 0.683 result = self.fn(self.args, *self.kwargs)

10 0.683 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

10 0.683 File "/usr/local/lib/python3.11/site-packages/datasette/database.py", line 209, in in_thread

10 0.683 self.ds._prepare_connection(conn, self.name)

10 0.683 File "/usr/local/lib/python3.11/site-packages/datasette/app.py", line 593, in _prepare_connection

10 0.683 conn.execute("SELECT load_extension(?)", [extension])

10 0.683 sqlite3.OperationalError: /usr/lib/x86_64-linux-gnu/mod_spatialite.so.so: cannot open shared object file: No such file or directory

executor failed running [/bin/sh -c datasette inspect nps-spatialite.db --inspect-file inspect-data.json]: exit code: 1 ```

The virtual table's _config version: 4 seems to indicate FTS5.

Codecov Report

Base: 92.55% // Head: 92.55% // No change to project coverage :thumbsup:

Coverage data is based on head (5eb4ea4) compared to base (00e233d). Patch has no changes to coverable lines.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

Codecov Report

Base: 92.55% // Head: 92.55% // No change to project coverage :thumbsup:

Coverage data is based on head (f254be4) compared to base (6f610e1). Patch has no changes to coverable lines.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

I appreciate your response @simonw - thanks!

I'll clarify what we need further - let's imagine we have 2000 SQLLite databases (for 2000 tenants), but we only want to run one datasette instance for each of those tenants to query/use datasette against their own database only. This means the "connection" between datasette and the SQLLite database would be dynamic, based on the tenantID that's required on an incoming request.

Is there any specific config or other considerations in this use case, to minimize memory use on a single, efficient VM and serve queries to all these tenants?

cc @muadham

Happy Birthday Datasette!

Thanks Simon!!

I use datasette on everything most notably my flickr metadata SQLite DB to make art.

Datasette lite on my 2019 flickr metadata is super helpful too: https://lite.datasette.io/?csv=https%3A%2F%2Fraw.githubusercontent.com%2Frtanglao%2Frt-flickr-sqlite-csv%2Fmain%2F2019-roland-flickr-metadata.csv

Even better datasette lite on all firefox support questions from 2021: https://lite.datasette.io/?url=https%3A%2F%2Fraw.githubusercontent.com%2Frtanglao%2Frt-kits-api3%2Fmain%2FYEARLY_CSV_FILES%2F2021-firefox-sumo-questions.db

Thanks again Simon! So great! What a gift to the world!!!!!!

I have no particular point, I just want to say being around for the 1.0 release of Datasette seems historic and legendary to witness.

Datasette usage comments for its 5th anniversary celebration:

I use Datasette and related tools for a Cosmology Researcher Talks database app project, which is described in the github Readme

The app hosted on the Google Cloud Run service also uses other Datasette-related tools developed by Simon - datasette-render-markdown, csvs-to-sqlite, datasette-template-sql, and datasette-block-robots. This is one of two apps used for querying the talks database, each has it pros/cons as described in the github Readme.

At present, over 170 different sites that host cosmology talks are scraped to collect new talks for import into the sqlite database. The shot-scraper and sqlite-utils tools are a major help for this.

I also use the Mastodon API to get my favorites, toots, and boosts into a local database so I can do searches on the data. This was done on Twitter and was then extended to the Mastodon data. Again, sqlite-utils is an important tool for this.

Happy Birthday Datasette!

I am a librarian at the Université du Québec à Montréal (UQAM) and I've been using Datasette to publish excerpts of our library data. There are several use cases I'm working with as a proof of concept : 1. New titles list : based on reports of recent acquisitions by subject, discipline, etc. 2. List of all UQAM theses and dissertations : based on an extract of bibliographic records 3. List of all publications by UQAM Authors : based on an extract of bibliographic records

See our prototype under construction here : https://datasette-bib.uqam.ca/ (some bits and pieces have been translated into French)

Datasette is amazing, there is so much potential here for libraries. Thanks to Simon and all the contributors for this outstanding effort. Also sqlite-utils deserves special mention as incredibly handy and useful.

I use Datasette to analyze blocklists by using csv-to-sqlite to pull their contents into a database and Datasette to look around through them. I also use its REST API to query said database as part of filtering out garbage from domains found in those blocklists.

Nothing spectacular yet but I think this falls under "cool/cute application of datasette": improving fakedata performance for fun. tl;dr I used datasette to visualize benchmarking data.

I'm a cryptic crossword enthusiast and have spent a lot of time scraping and parsing cryptic crossword clues from various blogs, forums and publications. The result is over half a million clues from cryptic crosswords over the past twelve years, including the clue, answer, puzzle date, puzzle name and a link to the original source. This is all hosted using Datasette, which has been a delight to use: https://cryptics.georgeho.org/

This dataset is a significant work of crossword archivism and scholarship, as acquiring historical crosswords and structuring their contents require focused effort and tedious cleaning that few are willing to do for such trivial data - for example, according to this 2004 selection guide, the Library of Congress explicitly does not collect crossword puzzles. Anecdotally, I know that many constructors/setters of cryptic crosswords use this dataset as a resource, and some even simply call it "the database" - this is probably one of the most impactful data projects I've worked on!

I work at The Wall Street Journal as a computational journalist and serve as our self-appointed Datasette evangelist. They say that to a hammer everything looks like a nail, but the reality is newsrooms find themselves in a sea of nails!

I've only got a couple public projects that I can share, but happy to offer you a look at some of the internal projects.

More often than not the internal projects stay internal because the reporting doesn't lead anywhere or I can't convince an editor to greenlight it. But imho that's the beauty of datasette: a (relatively) painless mechanism to see if there's any there there.

• WSJ Inflation Tracker
• I scraped the oscars website and turned it into a datasette instance and ran the numbers on best actress/best picture overlap

Hi Simon. I have just started experimenting with datasette in earnest, looking at it's suitability for air quality open data. A bulk upsert \ upsert_all would be very useful for me in enabling real time data to be pushed from a sql server database with FME server to a datasette db.

An hourly process queries the last 2 hours of data and pushes that to my database, inserting new data and updating existing combinations of pk siteid and date_time. This is already implemented on our current open data portal. Excited to see your progress with this! Thank you for this amazing software.

Amazing - thank you for fixing and releasing that so quickly and for showing your process! <3

I had the problem this morning when running:

Python==3.9.6 sqlite3.sqlite_version==3.37.0 sqlite-utils==3.30

I upgraded to: Python ==3.10.8 sqlite3.sqlite_version==3.37.2 sqlite-utils==3.30

and the error did not appear anymore.

Hope this helps Ryan

All seems to work well, but there are some glitches to do with proxies, see #1883 .

Excited to use this :)

The following worked for deployment to RStudio / Posit Connect

An app.py along the lines of:

```python from pathlib import Path

from datasette.app import Datasette

example_db = Path(file).parent / "data" / "example.db"

use connect 'Content URL' setting here to set app to /datasette/

ds = Datasette(files=[example_db], settings={"base_url": "/datasette/"})

ds._startup_invoked = True ds_app = ds.app() Then to deploy, from within a virtualenv with `rsconnect-python`sh rsconnect write-manifest fastapi -p $VIRTUAL_ENV/bin/python -e app:ds_app -o . rsconnect deploy manifest manifest.json -n <name of connect server> -t "Example Datasette" ```

Follow on question - is all memory use @simonw - for both datasette and SQLlite confined to the "query time" itself i.e. the memory use is relevant only to a particular transaction or query - and then subsequently released?

Codecov Report

Base: 96.25% // Head: 96.09% // Decreases project coverage by -0.15% :warning:

Coverage data is based on head (2d6a149) compared to base (529110e). Patch coverage: 63.63% of modified lines in pull request are covered.

:exclamation: Current head 2d6a149 differs from pull request most recent head 43a8c4c. Consider uploading reports for the commit 43a8c4c to get more accurate results

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

The Windows aspect is a red herring: OP's sample above produces the same error on Linux. (Though I don't know what's going on with the CI).

The same error can also be obtained by passing an io from a file opened in non-binary mode ('r' as opposed to 'rb') to rows_from_file(). This is how I got here.

The fix for my case is easy: open the file in mode 'rb'. The analagous fix for OP's problem also works: use BytesIO in place of StringIO.

Minimal test case (derived from utils.py):

``` python import io from typing import cast

fp = io.StringIO("id,name\n1,Cleo") # error

fp = io.BytesIO(bytes("id,name\n1,Cleo", encoding='utf-8')) # okay reader = io.BufferedReader(cast(io.RawIOBase, fp)) reader.peek(1) # exception thrown here `` I see the signature ofrows_from_file()correctly hasfp: BinaryIObut I guess you'd need either a runtime type check for that (not allios havemode()), or to catch theAttributeErroronpeek()` to produce a better error for users. Neither option is ideal.

Some thoughts on testing binary-ness of ios in this SO question: https://stackoverflow.com/questions/44584829/how-to-determine-if-file-is-opened-in-binary-or-text-mode

There’s also a limit of 3200 tweets. I wonder if that can be circumvented somehow.

With respect to the typing of Table class itself, my interim solution:

python from sqlite_utils.db import Table def tbl(self, table_name: str) -> Table: tbl = self.db[table_name] if isinstance(tbl, Table): return tbl raise Exception(f"Missing {table_name=}")

With respect to @chapmanjacobd concern on the DEFAULT being an empty class, have also been using # type: ignore, e.g.

python @classmethod def insert_list(cls, areas: list[str]): return meta.tbl(meta.Areas).insert_all( ({"area": a} for a in areas), ignore=True # type: ignore )

Codecov Report

Base: 92.55% // Head: 92.55% // No change to project coverage :thumbsup:

Coverage data is based on head (4faa4fd) compared to base (bf00b0b). Patch has no changes to coverable lines.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

Here is my suggestion:

^\s*((?:\-\-.*?\n\s*)|(?:/\*.*?(?=\*/)\*/\s*))*select\b

See the following test: https://regex101.com/r/Doeqqa/1

And here I played all your tests: https://regexr.com/713ir

Sorry I forgot the -- comments like that.

I'm afraid there is an issue in your regexp, see: https://regex101.com/r/pyubJf/1

I guess I can fix it.

I guess the issue is here: https://github.com/simonw/datasette/blob/9676b2deb07cff20247ba91dad3e84a4ab0b00d1/datasette/utils/init.py#L209

Here is a working regexp allowing it: diff - re.compile(r"^select\b"), + re.compile(r"^\s*(/\*.+?(?=\*/)\*/\s*)*select"), ^\s*: beginning by 0 or an infinite number of \s (spaces, tabs, newlines...) (/\*.+?(?=\*/)\*/\s*)*: 0 or an infinite number of chars beginning by /* and ending to the next occurrence of */ followed by 0 or an infinite number of \s

You can play with the regexp here: https://regex101.com/r/aESXDL/3

On that basis, I think the model described above where tokens mainly work to provide an "act on behalf of this actor" - but with optional additional constraints - is a good one.

This is what we do for Sandstorm essentially and I fully agree it's the right way to do API tokens in multiuser systems.

Constraints will definitely be important though. I know I want a token to submit error reports programmatically, but I wouldn't want that token to convey my right to delete tables and records, Little Bobby Tables is out there somewhere, and he's all grown up now.

How are you tying the bearer token to identity? I'm excited to see this feature, and since Sandstorm controls API access using the same header, it also will transparently support the API documentation here, but we strip the bearer before the request reaches the app (replacing it with our existing auth headers, of course).

Codecov Report

Base: 92.52% // Head: 92.54% // Increases project coverage by +0.02% :tada:

Coverage data is based on head (3623475) compared to base (79aa0de). Patch coverage: 100.00% of modified lines in pull request are covered.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

It was part of a larger idea I was exploring around ensuring Datasette could be used to start interacting with CSV/JSON data out-of-the-box, without needing to first convert that data into SQLite using separate tools.

This would be great. My organization deals with very nested JSON open data and I have been wanting to find a way to hook into datasette so that the analysts do not have to first convert to sqlite first.

This can kind of be done with datasette-lite.

From this random nested JSON API: https://api.nobelprize.org/v1/prize.json

You can use the API of https://flatterer.herokuapp.com to return a multi table sqlite database:

https://lite.datasette.io/?url=https://flatterer.herokuapp.com/api/convert?output_format=sqlite%26file_url=https://api.nobelprize.org/v1/prize.json

This is great and fun, but it would be great if there was some plugin mechanism that you could feed a local datasette a nested JSON file directly, possibly hooking into other flattening tools for this.

I think there's probably some core plugins a new user should just "get" maybe. I feel our Sandstorm package has really reasonable defaults, the ability to bring data in for instance. However, you also have to know that's in the hamburger menu, it's not necessarily intuitive where to start.

You could try using msys2. I've had better luck running python CLIs within that system on Windows.

Here is a guide: https://github.com/chapmanjacobd/lb/blob/main/Windows.md#prep

also, it says that authenticated requests have a much higher "rate limit". Unauthenticated requests only get 60 req/hour ?? seems more like a quota than a "rate limit" (although I guess that is semantic equivalence)

You would want to use x-ratelimit-reset

time.sleep(r['x-ratelimit-reset'] + 1 - time.time())

But a more complete solution would bring authenticated requests to the other subcommands. I'm surprised only github-to-sqlite get is using the --auth= CLI flag

Codecov Report

Base: 96.37% // Head: 96.25% // Decreases project coverage by -0.11% :warning:

Coverage data is based on head (a25e838) compared to base (d792dad). Patch coverage: 40.00% of modified lines in pull request are covered.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

Codecov Report

Base: 96.37% // Head: 96.37% // Increases project coverage by +0.00% :tada:

Coverage data is based on head (75577e5) compared to base (d792dad). Patch coverage: 100.00% of modified lines in pull request are covered.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

Codecov Report

Base: 92.50% // Head: 92.51% // Increases project coverage by +0.01% :tada:

Coverage data is based on head (9bead2a) compared to base (eff1124). Patch coverage: 100.00% of modified lines in pull request are covered.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

Ooh, I didn't even think about links in tables! You're definitely right on the approach to this. It might also be a really good "stupidly simple" plugin for me to try to build myself, which could be fun.

Codecov Report

Base: 92.50% // Head: 92.50% // No change to project coverage :thumbsup:

Coverage data is based on head (c12447e) compared to base (eff1124). Patch has no changes to coverable lines.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

Codecov Report

Base: 91.71% // Head: 92.50% // Increases project coverage by +0.78% :tada:

Coverage data is based on head (b4b92df) compared to base (cb1e093). Patch has no changes to coverable lines.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

Here is some working javascript code. There might be better solution, I'm not a JS expert. ```javascript var show_hide = document.querySelector(".show-hide-sql > a");

// Hide SQL query if the URL opened with #_hide_sql
var hash = window.location.hash;
if(hash === "#_hide_sql") {
    hide_sql();
}
show_hide.setAttribute("href", "#");
show_hide.addEventListener("click", toggle_sql_display);

function toggle_sql_display() {
    if (show_hide.innerText === "hide") {
        hide_sql();
        return;
    }
    if (show_hide.innerText === "show") {
        show_sql();
        return;
    }
}

function hide_sql() {
    sql_element.style.cssText="display:none";
    show_hide.innerHTML = "show";
    show_hide.setAttribute("href", "#_hide_sql");
}

function show_sql() {
    sql_element.style.cssText="display:block";
    show_hide.innerHTML = "hide";
    show_hide.setAttribute("href", "#_show_sql");
}

```

Codecov Report

Base: 91.58% // Head: 92.50% // Increases project coverage by +0.91% :tada:

Coverage data is based on head (b545b6a) compared to base (5f9f567). Patch coverage: 90.47% of modified lines in pull request are covered.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

@simonw rebased against latest, please let me know if i should drop this PR.

I'm tempted to add word-wrap: anywhere only to links that are know to be longer than a certain threshold.

Make sense IMHO.

Codecov Report

Base: 92.50% // Head: 92.50% // No change to project coverage :thumbsup:

Coverage data is based on head (1f0c557) compared to base (34defdc). Patch has no changes to coverable lines.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

https://github.com/simonw/datasette-cors seems to workaround this

I'm glad the bug report served some purpose. Frankly I just needed the method signature, that is why the documentation you mention wasn't read.

On Tue, Sep 27, 2022, 9:05 PM Simon Willison @.***> wrote:

Though now I notice that the copy right there needs to be updated to reflect the new row parameter to render_cell!

— Reply to this email directly, view it on GitHub https://github.com/simonw/datasette/issues/1826#issuecomment-1260357878, or unsubscribe https://github.com/notifications/unsubscribe-auth/AP46PCLI6KVDQTFLWVZODRLWAO72JANCNFSM6AAAAAAQXKWIJA . You are receiving this because you authored the thread.Message ID: @.***>

Codecov Report

Base: 91.58% // Head: 91.58% // No change to project coverage :thumbsup:

Coverage data is based on head (b16eb2f) compared to base (5f9f567). Patch has no changes to coverable lines.

:exclamation: Current head b16eb2f differs from pull request most recent head e7e96dc. Consider uploading reports for the commit e7e96dc to get more accurate results

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

thoughts on order of precedence to use:

• sqlite-utils count, if present. closest thing to a standard i guess.
• row(max_id) if like, the first and/or last x amount of rows ids are all contiguous. kind of a cheap/dumb/imperfect heuristic to see if the table is dump/not dump. if the check passes, still stick on est. after the display.
• count(*) if enabled in datasette

I wonder if having an option for displaying the max row id might help too. Not accurate especially if something was deleted, but useful for DBs as a dump.

HI Simon,

looks good, I noticed you wanted to use a regex to detect it, you might be interested in github.com/iafisher/sqliteparser which creates an ast of the create table statement, not every option supported yet but i forked it and am adding all the possible options in a create table (and create index) statement.

For completeness, the regex requires a bit more dark magic to capture the following lines, here is a working expression: https://regex101.com/r/rsuEcs/1

``` sqlite-utils insert /tmp/log.db log multiline.log --text --convert " import re

r = re.compile(r'^(?P<datetime>\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2})(?:\:\s)(?P<log>(.\s\s.|.*)+)', re.MULTILINE)

def convert(text): return [m.groupdict() for m in r.finditer(text)] " ```

BEGIN TRANSACTION; CREATE TABLE [log] ( [datetime] TEXT, [log] TEXT ); INSERT INTO "log" VALUES('2022-03-01T12:04:52','Here is a log message that spans multiple lines'); INSERT INTO "log" VALUES('2022-03-01T12:04:52','This is a single line'); INSERT INTO "log" VALUES('2022-03-01T12:04:52','Here is another message that spans multiple lines'); COMMIT;

🤯 this is beautiful. Thanks @simonw !

This would be great! I just went through the process of figuring out the minimum permissions for a service account to run datasette publish cloudrun for PUDL's datasette deployment. These are the roles I gave the service account (disclaim: I'm not sure these are the minimum permissions):

• Cloud Build Service Account: The SA needs this role to publish the build on Cloud Build.
• Cloud Run Admin for the Cloud Run datasette service so the SA can deploy the build.
• I gave the SA the Storage Admin role on the bucket Cloud Build creates to store the build tar files.
• The Viewer Role is required for storing build logs in the default bucket. More on this below!

The Viewer Role is a Basic IAM role that Google does not recommend using:

Caution: Basic roles include thousands of permissions across all Google Cloud services. In production environments, do not grant basic roles unless there is no alternative. Instead, grant the most limited predefined roles or custom roles that meet your needs.

If you don't grant the Viewer role the gcloud builds submit command will successfully create a build but returns exit code 1, preventing the script from getting to the cloud run step:

``` ERROR: (gcloud.builds.submit) The build is running, and logs are being written to the default logs bucket. This tool can only stream logs if you are Viewer/Owner of the project and, if applicable, allowed by your VPC-SC security policy.

The default logs bucket is always outside any VPC-SC security perimeter. If you want your logs saved inside your VPC-SC perimeter, use your own bucket. See https://cloud.google.com/build/docs/securing-builds/store-manage-build-logs. long stack trace... CalledProcessError: Command 'gcloud builds submit --tag gcr.io/catalyst-cooperative-pudl/datasette' returned non-zero exit status 1. ```

You can store Cloud Build logs in a user-created bucket which only requires the Storage Admin role. However, you have to pass a config file to gcloud builds submit, which isn't possible with the current options for datasette publish cloudrun.

I propose we add an additional CLI option to datasette publish cloudrun called --build-config that allows users to pass a config file specifying a user create Cloud Build log bucket.

yay! Thanks a bunch for the twitter-to-sqlite friends command!

The twitter "Download an archive of your data" feature doesn't include who I follow, so this is particularly handy.

The whole Dogsheep thing is great :) I've written about similar things under cloud-services: - 2021: Closet Librarian Approach to Cloud Services - 2015: jukekb - Browse iTunes libraries and upload playlists to Google Music

Codecov Report

Base: 91.70% // Head: 91.70% // No change to project coverage :thumbsup:

Coverage data is based on head (b3855e7) compared to base (b40872f). Patch has no changes to coverable lines.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

Codecov Report

Base: 96.47% // Head: 96.52% // Increases project coverage by +0.04% :tada:

Coverage data is based on head (0acbc68) compared to base (d9b9e07). Patch coverage: 100.00% of modified lines in pull request are covered.

:exclamation: Current head 0acbc68 differs from pull request most recent head d5db749. Consider uploading reports for the commit d5db749 to get more accurate results

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

I was wondering if you have any more thoughts on this? I have a tangible use case now: adding a "vector" column to a database to support semantic search using doc2vec embeddings (example; note that the vtfunc package may no longer be reliable...).

Perfect. Apologies, I searched but didn’t see that one for some reason. Thank you.

On Fri, Sep 9, 2022 at 3:04 PM Simon Willison @.***> wrote:

This isn't supported yet - there's an older issue for that here:

• 117 https://github.com/simonw/sqlite-utils/issues/117

— Reply to this email directly, view it on GitHub https://github.com/simonw/sqlite-utils/issues/487#issuecomment-1242409766, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAEECKBODHZCKWTN5OO6RI3V5OJ5RANCNFSM6AAAAAAQIV3YIU . You are receiving this because you authored the thread.Message ID: @.***>

@simonw Will upload_csvs treat that as a database it can import to if you set it as mutable?

Thanks!

Will do!

Perhaps some ways to address this.

1. Add a horizontal scrollbar at the top of the table. There are some solutions here: https://stackoverflow.com/questions/3934271/horizontal-scrollbar-on-top-and-bottom-of-table
2. Use a fixed table header. It would be useful when you're lost in the middle of a very big table. Pure CSS solutions seem to exist: https://stackoverflow.com/questions/21168521/table-fixed-header-and-scrollable-body
3. Maybe a possibility to resize columns. Not sure about that because it would more work not to lose it after each reload.
4. A way to keep favorite views for each user. The process would be: I select the column I want or not (with existing "settings" icon of each column); then I select a "favoritize view" option somewhere; then I can recall all my favorite views from a menu. These data could be hosted on the browser.

Tested PR just now in private org, fetched >2k repos infos flawlessly!

poetry run github-to-sqlite repos --organization github.db MYORG

Codecov Report

Base: 91.66% // Head: 91.67% // Increases project coverage by +0.01% :tada:

Coverage data is based on head (2db6f9c) compared to base (51030df). Patch coverage: 100.00% of modified lines in pull request are covered.

:exclamation: Current head 2db6f9c differs from pull request most recent head 7b6a2b4. Consider uploading reports for the commit 7b6a2b4 to get more accurate results

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

Codecov Report

Base: 91.66% // Head: 91.66% // No change to project coverage :thumbsup:

Coverage data is based on head (3f24edd) compared to base (ba35105). Patch has no changes to coverable lines.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

I was looking for something like this today, for extracting columns containing objects (and arrays of objects) into separate tables.

Would it make sense (especially for the fields containing arrays of objects) to create a one-to-many relationship, where each row of the newly created table would contain the id of the row that originally contained it?

If the extracted objects have a unique id and are repeated, it could even create a many-to-many relationship, with a third table for the joins.

@zschira is working with Pydantic while converting between and validating JSON frictionless datapackage descriptors that annotate an SQLite DB (extracted from FERC's XBRL data) and the Datasette YAML metadata so we can publish them with Datasette. Maybe there's some overlap? We've been loving Pydantic.

I want to do may be 3 or 4 PR for your evaluation, and when you allowed then I want supporting that effort

El mié, 31 de ago. de 2022, 10:36 p. m., Simon Willison ***@***.*** escribió:

I don't want to start any efforts around documentation translation until after the Datasette 1.0 release, because I'd like to be confident that we're not translating documentation that may have some big changes before Datasette is fully stable!

— Reply to this email directly, view it on GitHub https://github.com/simonw/datasette/issues/1775#issuecomment-1233697166, or unsubscribe https://github.com/notifications/unsubscribe-auth/AADIWFDUWQ22P3KVJVMRC7TV4AQFNANCNFSM55EFFS5Q . You are receiving this because you authored the thread.Message ID: @.***>

I want to begin translation to es and it documentation, if u like i would do PR asap. But user interface, frontend and backend is a good feature be i18n support. When you have may be .pot file I can translate to, or Json file with all strings for internationalized

I want contribute if strings is in pot file, or Json, yml, yaml, Js or other type ..

I also would like a French version UI and am prepared to contribute to this translation.

Having worked an i18n project in the past, there can be complexities with the strings and the various grammars and syntaxes, depending on how things are set up. It's a pretty big job to retroactively i18n-ify things, but for us at least we have requirements to provide French-language interfaces at our University, so I would be willing to participate in this initiative.

Codecov Report

Base: 96.47% // Head: 96.47% // Increases project coverage by +0.00% :tada:

Coverage data is based on head (3dbfca6) compared to base (087753c). Patch coverage: 100.00% of modified lines in pull request are covered.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

Creating the table before inserting is a useful workaround, thanks. It does require figuring out the create table syntax and listing all the fields manually, though, which loses some of the magic of sqlite-utils.

I was expecting to find an option like --headers=foo,bar (or --header-row='foo\tbar', if that would be easier) - not necessarily that exact syntax, but something that would essentially be treated the same as having a header row in the file.

Is it still nfortunately slow and tricky when playing with floats ?

Codecov Report

Merging #468 (fc38480) into main (c5f8a2e) will decrease coverage by 0.12%. The diff coverage is 88.88%.

:exclamation: Current head fc38480 differs from pull request most recent head 2f6a64f. Consider uploading reports for the commit 2f6a64f to get more accurate results

```diff @@ Coverage Diff @@

main #468 +/-

========================================== - Coverage 96.60% 96.47% -0.13%
========================================== Files 6 6
Lines 2590 2640 +50
========================================== + Hits 2502 2547 +45
- Misses 88 93 +5
```

| Impacted Files | Coverage Δ | | |---|---|---| | sqlite_utils/cli.py | 95.86% <66.66%> (-0.09%) | :arrow_down: | | sqlite_utils/db.py | 97.31% <90.19%> (-0.23%) | :arrow_down: |

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

Codecov Report

Merging #473 (72db599) into main (a46a5e3) will decrease coverage by 0.07%. The diff coverage is 50.00%.

```diff @@ Coverage Diff @@

main #473 +/-

========================================== - Coverage 96.67% 96.60% -0.08%
========================================== Files 6 6
Lines 2587 2590 +3
========================================== + Hits 2501 2502 +1
- Misses 86 88 +2
```

| Impacted Files | Coverage Δ | | |---|---|---| | sqlite_utils/cli.py | 95.94% <50.00%> (-0.18%) | :arrow_down: |

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

Codecov Report

Merging #1792 (f61cfb7) into main (fd1086c) will increase coverage by 0.04%. The diff coverage is n/a.

```diff @@ Coverage Diff @@

main #1792 +/-

========================================== + Coverage 91.66% 91.71% +0.04%
========================================== Files 38 38
Lines 4705 4705
========================================== + Hits 4313 4315 +2
+ Misses 392 390 -2
```

| Impacted Files | Coverage Δ | | |---|---|---| | datasette/database.py | 94.13% <0.00%> (+0.65%) | :arrow_up: |

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

For now I've installed datasette via homebrew.

I then converted my CSV to SQLite via sqlite-utils as per the article Cleaning data with sqlite.

Now I can use my metadata.json file as follows:

datasette mydb.db -m mymetadata.json

This bug affects me as well. Env:

Python 3.8.12 sqlite-utils, version 3.28 sqlite3 3.32.3 MacOS Big Sur 11.6.7 Intel

Similar to @mdrovdahl, I was able to work around this bug by piping the SQL string constructed in add_foreign_keys to the sqlite3 command itself. Specifically, if you're trying to patch this yourself, replace lines 1026-1039 of db.py in your site packages with something similar to the following:

print("PRAGMA writable_schema = 1;") for table_name, new_sql in table_sql.items(): print("UPDATE sqlite_master SET sql = '{}' WHERE name = '{}';".format( new_sql, table_name) ) print("PRAGMA writable_schema = 0;") print("VACUUM;")

Then from your terminal: db-to-sqlite "<connection string>" your.db --all > output.sql && sqlite3 your.db < output.sql

If you want to run this with -p, you'll have to actually open a file in code to write to instead of redirecting the output.

I didn't see there is a PR about this: https://github.com/dogsheep/pocket-to-sqlite/pull/7

Codecov Report

Merging #465 (bb90d33) into main (45e24de) will not change coverage. The diff coverage is n/a.

```diff @@ Coverage Diff @@

main #465 +/-

======================================= Coverage 96.64% 96.64%
======================================= Files 6 6
Lines 2566 2566
======================================= Hits 2480 2480
Misses 86 86
```

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

closing as this is now the default

I've been looking into how to to get this data out of Github (especially now there are "secondary rate limits" without an advertised allowance separate from the regular rate limits.

I've had decent success with the Airbyte github extractor (aside from one data quality issue https://github.com/airbytehq/airbyte/pull/15420 ). Airbyte splits data extraction between the GraphQL and REST endpoints depending on the resource type, but they're very comprehensive.

https://github.com/airbytehq/airbyte/blob/306a75ef5370728e0912cf52a1a898a530db0c90/airbyte-integrations/connectors/source-github/source_github/streams.py#L22-L122

Before this, I tried a few solutions in my own custom wrapper mentioned in this thread + its children https://github.com/PyGithub/PyGithub/issues/1989 , but they weren't working as expected.

I am getting the same error when using github-to-sqlite (which uses sqlite-utils internally). I am also using an M1 MacBook Pro, with macOS Monterey 12.5, with Python 3.10.6 for arm64 installed using pyenv. I have sqlite-utils 3.28 installed.

Codecov Report

Merging #462 (0a14a03) into main (1856002) will not change coverage. The diff coverage is n/a.

```diff @@ Coverage Diff @@

main #462 +/-

======================================= Coverage 96.64% 96.64%
======================================= Files 6 6
Lines 2566 2566
======================================= Hits 2480 2480
Misses 86 86
```

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 1856002...0a14a03. Read the comment docs.

Codecov Report

Merging #460 (f21f4cc) into main (1856002) will not change coverage. The diff coverage is n/a.

```diff @@ Coverage Diff @@

main #460 +/-

======================================= Coverage 96.64% 96.64%
======================================= Files 6 6
Lines 2566 2566
======================================= Hits 2480 2480
Misses 86 86
```

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 1856002...f21f4cc. Read the comment docs.

test_dockerfile.py

I'm skipping this for the moment because the new Dockerfile shape introduced in #1249 (comment) isn't compatible with this technique, since it installs Datasette from PyPI rather than directly from the repo.

Will need to change that if I want to do this unit tests thing.

Hello, can't you copy in a later step directly in the output directory, e.g. COPY test_dockerfile.py /usr/lib/python*/.. or something like that ?

I was able to fight through this by capturing the SQL commands from the add_foreign_keys() function in sqlite-utils and then executing them manually via the sqlite3 client, first setting PRAGMA writable_schema on and then updating the sqlite_master table. Still no clue why they were failing when run in context...

Codecov Report

Merging #457 (80b3907) into main (9e6ccea) will not change coverage. The diff coverage is n/a.

```diff @@ Coverage Diff @@

main #457 +/-

======================================= Coverage 96.64% 96.64%
======================================= Files 6 6
Lines 2566 2566
======================================= Hits 2480 2480
Misses 86 86
```

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 9e6ccea...80b3907. Read the comment docs.

I'm using this in a separate FastAPI app, worked perfectly when I changed the AsyncBoundLogger to BoundLogger only.

Also for some reason, I'm now getting some logs surfacing from internal packages, like Elasticsearch. But don't have time to deal with that now.

👋 Any update on getting this merged?

Alternatively, is there a work around for this issue to unblock myself?

edit to add: huge fan of both this project and osxphotos, thanks so much for your work here 🙏 If I had any experience with Python I would offer to help but somehow I've managed to not write any Python in 10+ years of programming 😅

Edit again to add:

Alternatively, is there a work around for this issue to unblock myself?

Yes, there is. I was able to apply the patch of this PR and it applies (mostly) cleanly and works.

• verified I have a high enough version of osxphotos
• downloaded the .patch of this (by appending .patch to the URL
• edited the patch to remove the setup.py changes
• cd to the directory containing dogsheep-photos and git apply 31.patch

hadley wickham's melt and reshape could be good inspo: http://had.co.nz/reshape/introduction.pdf

Note that Hadley has since implemented pivot_longer and pivot_wider instead of the previous verbs/functions that he used. Those can be found in the tidyr package and are probably the best reference which includes all of the learnings from years of user feedback. https://tidyr.tidyverse.org/articles/pivot.html

Codecov Report

Merging #455 (1b35a92) into main (855bce8) will not change coverage. The diff coverage is n/a.

```diff @@ Coverage Diff @@

main #455 +/-

======================================= Coverage 96.64% 96.64%
======================================= Files 6 6
Lines 2566 2566
======================================= Hits 2480 2480
Misses 86 86
```

| Impacted Files | Coverage Δ | | |---|---|---| | sqlite_utils/db.py | 97.54% <ø> (ø) | |

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 855bce8...1b35a92. Read the comment docs.