While I'm doing this I should rename this internal variable to avoid confusion in the future:

https://github.com/simonw/datasette/blob/e837095ef35ae155b4c78cc9a8b7133a48c94f03/datasette/app.py#L203

_Originally posted by @simonw in https://github.com/simonw/datasette/issues/1431#issuecomment-898072940_

For example:

• Go to https://cryptics.eigenfoo.xyz/clues/clues?_next=100 (this is the second page of results in a Datasette site)
• Search anything using the FTS search bar. For example, searching for hello will take you to https://cryptics.eigenfoo.xyz/clues/clues?_search=hello&_sort=rowid&_next=100
• A 500 Error: list index out of range is raised.

This is because the search URL includes the &_next=100 UTM parameter, carried over from where the FTS search was run. However, there isn't a second page in the search results, so a list index out of range error is raised. You can confirm that removing this UTM parameter from the URL returns the appropriate search results.

The FTS search request should strip any _next UTM parameter.

datasette, version 0.58.1
sqlite-utils, version 3.17

Updates the requirements on pytest-asyncio to permit the latest version.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

This PR adds the Access-Control-Allow-Headers flag when CORS mode is enabled.

This would fix https://github.com/simonw/datasette-auth-tokens/issues/4. When making cross-origin requests, the server must respond with all allowable HTTP headers. A Datasette instance using auth tokens must accept the Authorization HTTP header in order for cross-origin authenticated requests to take place.

Please let me know if there's a better way of doing this! I couldn't figure out a way to change the app's response from the plugin itself, so I'm starting here. If you'd rather this logic live in the plugin, I'd love any guidance you're able to give.

👋 trying to upgrade the formula to use the latest release, but runs into some regression test issue with --get command.

My QQ is does this datasette --get '/:memory:.json?sql=select+3*5' supposed to return 15? Thanks!

relates to https://github.com/Homebrew/homebrew-core/pull/87369

currently --flatten option works only for insert command, it would be cool if it worked for memory as well to query nested json

I had to do this with a jq recipe today: https://til.simonwillison.net/cloudrun/tailing-cloud-run-request-logs

cat log.json | jq -c '[leaf_paths as $path | {
  "key": $path | join("_"), "value": getpath($path)
}] | from_entries' \
| sqlite-utils insert /tmp/logs.db logs - --nl --alter --batch-size 1

That was to turn something like this:

  {
    "httpRequest": {
      "latency": "0.112114537s",
      "requestMethod": "GET",
      "requestSize": "534",
      "status": 200,
    },
    "insertId": "6111722f000b5b4c4d4071e2",
    "labels": {
      "service": "datasette-io"
    }
  }

Into this instead:

{
    "httpRequest_latency": "0.112114537s",
    "httpRequest_requestMethod": "GET",
    "httpRequest_requestSize": "534",
    "httpRequest_status": 200,
    "insertId": "6111722f000b5b4c4d4071e2",
    "labels_service": "datasette-io"
}

I have to do this often enough that I think it should be an option, --flatten - so I can do this instead:

cat log.json | sqlite-utils insert /tmp/logs.db logs - --flatten

Per https://github.com/simonw/datasette/issues/1457, I was confused by the current docs and took a few minutes to sort out what the right combination of locations was.

This is a PR that differentiates the docs to split out /static/ in URL from --static option and ./static/ path.

Not wedded to the details in any way, happy to change to suit.

Fixes #1457.

Python 3.9.5
mypy 0.910
sqlite-utils 3.17.1

While using sqlite-utils as a library, when I use mypy for static type checking, it throws an error:

mypy .
src/etl.py:5: error: Skipping analyzing "sqlite_utils": found module but no type hints or library stubs
    import sqlite_utils
    ^
src/etl.py:5: note: See https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports
test/test_etl.py:4: error: Skipping analyzing "sqlite_utils": found module but no type hints or library stubs
    import sqlite_utils
    ^
Found 2 errors in 2 files (checked 7 source files)

When I add a py.typed file to the sqlite-utils package to mark it as PEP 561 compatible, the error goes away.

al@nbal ..b/python3.9/site-packages/sqlite_utils (git)-[main] % la
total 200
drwx------   3 al al   4096 Oct 14 22:00 .
drwx------ 117 al al   4096 Oct 12 21:12 ..
-rw-------   1 al al  64409 Oct 12 21:11 cli.py
-rw-------   1 al al 109092 Oct 12 21:11 db.py
-rw-------   1 al al      0 Oct 14 22:00 py.typed
-rw-------   1 al al    684 Oct 12 21:11 recipes.py
-rw-------   1 al al   7988 Oct 12 21:11 utils.py
-rw-------   1 al al    113 Oct 12 21:11 __init__.py

I would like to suggest adding a py.typed file to the repository.

See also the mypy docs on creating PEP 561 compatible packages:
https://mypy.readthedocs.io/en/stable/installed_packages.html#creating-pep-561-compatible-packages

Over in https://docs.datasette.io/en/stable/custom_templates.html#serving-static-files, there is the slightly comical example command -

datasette -m metadata.json --static static:static/ --memory

(now, with MORE STATIC!)

It took me a while to sort out all the URLs and paths involved because I wasn't being very clever. But in the interests of simplification and distinction, I might suggest something like

datasette -m metadata.json --static loc:static-files/ --memory

I will submit a PR for your consideration.

Updates the requirements on pyyaml to permit the latest version.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

This is caused by a change made to httpx in https://github.com/encode/httpx/releases/tag/0.20.0

Refs #1488

Updates the requirements on beautifulsoup4 to permit the latest version.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

This tests against the latest 3.10 RC, https://www.python.org/downloads/release/python-3100rc2/

Bumps black from 21.7b0 to 21.9b0.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

e.g. on https://covid-19.datasettes.com/covid/economist_excess_deaths

https://user-images.githubusercontent.com/9599/133025772-beaf055e-374f-4180-aad3-bb0215e8aaf3.png">

But if you add ?_facet=country to the URL that goes away: https://covid-19.datasettes.com/covid/economist_excess_deaths?_facet_size=5&_facet=country

The logic that decides if the "Facet by this" item is shown does not take default metadata.json facets into account.

Updates the requirements on pluggy to permit the latest version.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Change "above" to "below" to correct correspondence of reference to example.

Updates the requirements on pytest-xdist to permit the latest version.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Updates the requirements on pytest-timeout to permit the latest version.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

The datasette installation instructions discuss how to install with pipx, how to upgrade with pipx, and how to upgrade plugins with pipx but do not mention how to install a plugin with pipx. You discussed this on your blog but looks like this didn't make it in when you updated the docs for pipx (#756).

I'll submit a PR shortly to fix this.

Adds missing instructions for installing plugins via pipx

Reported by @eigenfoo in https://github.com/simonw/datasette/issues/1470

I tried to use https://your-foursquare-oauth-token.glitch.me/ to get my Swarm auth token and got a 403 after I clicked the Allow button:

I'm not sure if this is the right repo to report this in

Hi,

I needed to extract data in JSON Arrays to normalize data imports. I've quickly hacked the following together based on #241 which refers to #239 where you, @simonw, wrote:

Could this handle lists of objects too? That would be pretty amazing - if the column has a [{...}, {...}] list in it could turn that into a many-to-many.

They way this works in my work is that many-to-many relationships are created for anything that maps to an dictionary in a list, and many-to-one relations for everything else (assumed to be scalar values). Not sure what the best approach here would be? Are many-to-one relationships are at all useful here?

What do you think about this approach? I could try to add it to the cli interface and documentation if wanted.

Thanks for this awesome piece of software in any case! :sun_with_face:

I started work on this in #1481 where I found a Python 3.10 bug that needs a workaround in Janus, see:

• https://github.com/aio-libs/janus/issues/358

This is a tracking issue for anything else that shows up.

When I try to deploy a 4.8G SQLite database to Google Cloud Run, I get this error message:

Memory limit of 8192M exceeded with 8826M used. Consider increasing the memory limit, see https://cloud.google.com/run/docs/configuring/memory-limits

Unfortunately, the maximum amount of memory that can be allocated to an instance is 8192M.

Naively profiling the memory usage of running Datasette with this database locally on my MacBook shows the following memory usage (using Activity Monitor) when I just start up Datasette locally:

• Real Memory Size: 70.6 MB
• Virtual Memory Size: 4.51 GB
• Shared Memory Size: 2.5 MB
• Private Memory Size: 57.4 MB

I'm trying to understand if there's a query or other operation that gets run during container deployment that causes memory use to be so large and if this can be avoided somehow.

This is somewhat related to #1082, but on a different platform, so I decided to open a new issue.

I unfortunately was not successful to deploy to fly.io. Please see the details above of the three scenarios that I took. I am also new to datasette.

Failed to deploy. Attaching logs:
1. Tried with an app created via flyctl apps create frosty-fog-8565 and the ran datasette publish fly covid.db --app frosty-fog-8565

Deploying frosty-fog-8565
==> Validating app configuration
--> Validating app configuration done
Services
TCP 80/443 ⇢ 8080

Error error connecting to docker: An unknown error occured.

Traceback (most recent call last):
  File "c:\users\grott\anaconda3\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "c:\users\grott\anaconda3\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Users\grott\Anaconda3\Scripts\datasette.exe\__main__.py", line 7, in <module>
  File "c:\users\grott\anaconda3\lib\site-packages\click\core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "c:\users\grott\anaconda3\lib\site-packages\click\core.py", line 782, in main
    rv = self.invoke(ctx)
  File "c:\users\grott\anaconda3\lib\site-packages\click\core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "c:\users\grott\anaconda3\lib\site-packages\click\core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "c:\users\grott\anaconda3\lib\site-packages\click\core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "c:\users\grott\anaconda3\lib\site-packages\click\core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "c:\users\grott\anaconda3\lib\site-packages\datasette_publish_fly\__init__.py", line 156, in fly
    "--remote-only",
  File "c:\users\grott\anaconda3\lib\contextlib.py", line 119, in __exit__
    next(self.gen)
  File "c:\users\grott\anaconda3\lib\site-packages\datasette\utils\__init__.py", line 451, in temporary_docker_directory
    tmp.cleanup()
  File "c:\users\grott\anaconda3\lib\tempfile.py", line 811, in cleanup
    _shutil.rmtree(self.name)
  File "c:\users\grott\anaconda3\lib\shutil.py", line 516, in rmtree
    return _rmtree_unsafe(path, onerror)
  File "c:\users\grott\anaconda3\lib\shutil.py", line 395, in _rmtree_unsafe
    _rmtree_unsafe(fullname, onerror)
  File "c:\users\grott\anaconda3\lib\shutil.py", line 404, in _rmtree_unsafe
    onerror(os.rmdir, path, sys.exc_info())
  File "c:\users\grott\anaconda3\lib\shutil.py", line 402, in _rmtree_unsafe
    os.rmdir(path)
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\grott\\AppData\\Local\\Temp\\tmpgcm8cz66\\frosty-fog-8565'

1. Tried also with an app that gets autogenerate when running flyctl launch. This also generates the .toml file. Ran then datasette publish fly covid.db --app dark-feather-168 but different error now
   ```Deploying dark-feather-168
   ==> Validating app configuration

Error not possible to validate configuration: server returned Post "https://api.fly.io/graphql": unexpected EOF

Traceback (most recent call last):
File "c:\users\grott\anaconda3\lib\runpy.py", line 193, in _run_module_as_main
"main", mod_spec)
exec(code, run_globals)
File "C:\Users\grott\Anaconda3\Scripts\datasette.exe__main__.py", line 7, in <module>
File "c:\users\grott\anaconda3\lib\site-packages\click\core.py", line 829, in call
return self.main(args, kwargs)
File "c:\users\grott\anaconda3\lib\site-packages\click\core.py", line 782, in main
rv = self.invoke(ctx)
File "c:\users\grott\anaconda3\lib\site-packages\click\core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "c:\users\grott\anaconda3\lib\site-packages\click\core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "c:\users\grott\anaconda3\lib\site-packages\click\core.py", line 1066, in invoke
return ctx.invoke(self.callback, ctx.params)
File "c:\users\grott\anaconda3\lib\site-packages\click\core.py", line 610, in invoke
return callback(args, **kwargs)
File "c:\users\grott\anaconda3\lib\site-packages\datasette_publish_fly__init__.py", line 156, in fly
"--remote-only",
File "c:\users\grott\anaconda3\lib\contextlib.py", line 119, in exit
next(self.gen)
File "c:\users\grott\anaconda3\lib\site-packages\datasette\utils__init__.py", line 451, in temporary_docker_directory
tmp.cleanup()
File "c:\users\grott\anaconda3\lib\tempfile.py", line 811, in cleanup
_shutil.rmtree(self.name)
File "c:\users\grott\anaconda3\lib\shutil.py", line 516, in rmtree
return _rmtree_unsafe(path, onerror)
File "c:\users\grott\anaconda3\lib\shutil.py", line 395, in _rmtree_unsafe
_rmtree_unsafe(fullname, onerror)
File "c:\users\grott\anaconda3\lib\shutil.py", line 404, in _rmtree_unsafe
onerror(os.rmdir, path, sys.exc_info())
File "c:\users\grott\anaconda3\lib\shutil.py", line 402, in _rmtree_unsafe
os.rmdir(path)
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\Users\grott\AppData\Local\Temp\tmpnoyewcre\dark-feather-168'

These are also the contents of the generated **.toml file** in 2 scenario:

fly.toml file generated for dark-feather-168 on 2021-09-28T20:35:44+02:00

app = "dark-feather-168"

kill_signal = "SIGINT"
kill_timeout = 5
processes = []

[env]

[experimental]
allowed_public_ports = []
auto_rollback = true

[[services]]
http_checks = []
internal_port = 8080
processes = ["app"]
protocol = "tcp"
script_checks = []

[services.concurrency]
hard_limit = 25
soft_limit = 20
type = "connections"

[[services.ports]]
handlers = ["http"]
port = 80

[[services.ports]]
handlers = ["tls", "http"]
port = 443

[[services.tcp_checks]]
grace_period = "1s"
interval = "15s"
restart_limit = 6
timeout = "2s"
```

1. But also trying datasette package covid.db to create a local DOCKERFILE to later try to push it via flyctl deploy fails as well.

```[+] Building 147.3s (11/11) FINISHED
=> [internal] load build definition from Dockerfile 0.2s
=> => transferring dockerfile: 396B 0.0s
=> [internal] load .dockerignore 0.1s
=> => transferring context: 2B 0.0s
=> [internal] load metadata for docker.io/library/python:3.8 4.7s
=> [auth] library/python:pull token for registry-1.docker.io 0.0s
=> [internal] load build context 0.1s
=> => transferring context: 82.37kB 0.0s
=> [1/5] FROM docker.io/library/python:3.8@sha256:530de807b46a11734e2587a784573c12c5034f2f14025f838589e6c0e3 108.3s
=> => resolve docker.io/library/python:3.8@sha256:530de807b46a11734e2587a784573c12c5034f2f14025f838589e6c0e3b5 0.0s
=> => sha256:56182bcdf4d4283aa1f46944b4ef7ac881e28b4d5526720a4e9ba03a4730846a 2.22kB / 2.22kB 0.0s
=> => sha256:955615a668ce169f8a1443fc6b6e6215f43fe0babfb4790712a2d3171f34d366 54.93MB / 54.93MB 21.6s
=> => sha256:911ea9f2bd51e53a455297e0631e18a72a86d7e2c8e1807176e80f991bde5d64 10.87MB / 10.87MB 15.5s
=> => sha256:530de807b46a11734e2587a784573c12c5034f2f14025f838589e6c0e3b5c5b6 1.86kB / 1.86kB 0.0s
=> => sha256:ff08f08727e50193dcf499afc30594c47e70cc96f6fcfd1a01240524624264d0 8.65kB / 8.65kB 0.0s
=> => sha256:2756ef5f69a5190f4308619e0f446d95f5515eef4a814dbad0bcebbbbc7b25a8 5.15MB / 5.15MB 6.4s
=> => sha256:27b0a22ee906271a6ce9ddd1754fdd7d3b59078e0b57b6cc054c7ed7ac301587 54.57MB / 54.57MB 37.7s
=> => sha256:8584d51a9262f9a3a436dea09ba40fa50f85802018f9bd299eee1bf538481077 196.45MB / 196.45MB 82.3s
=> => sha256:524774b7d3638702fe9ae0ea3fcfb81b027dfd75cc2fc14f0119e764b9543d58 6.29MB / 6.29MB 26.6s
=> => extracting sha256:955615a668ce169f8a1443fc6b6e6215f43fe0babfb4790712a2d3171f34d366 5.4s
=> => sha256:9460f6b75036e38367e2f27bb15e85777c5d6cd52ad168741c9566186415aa26 16.81MB / 16.81MB 40.5s
=> => extracting sha256:2756ef5f69a5190f4308619e0f446d95f5515eef4a814dbad0bcebbbbc7b25a8 0.6s
=> => extracting sha256:911ea9f2bd51e53a455297e0631e18a72a86d7e2c8e1807176e80f991bde5d64 0.6s
=> => sha256:9bc548096c181514aa1253966a330134d939496027f92f57ab376cd236eb280b 232B / 232B 40.1s
=> => extracting sha256:27b0a22ee906271a6ce9ddd1754fdd7d3b59078e0b57b6cc054c7ed7ac301587 5.8s
=> => sha256:1d87379b86b89fd3b8bb1621128f00c8f962756e6aaaed264ec38db733273543 2.35MB / 2.35MB 41.8s
=> => extracting sha256:8584d51a9262f9a3a436dea09ba40fa50f85802018f9bd299eee1bf538481077 18.8s
=> => extracting sha256:524774b7d3638702fe9ae0ea3fcfb81b027dfd75cc2fc14f0119e764b9543d58 1.2s
=> => extracting sha256:9460f6b75036e38367e2f27bb15e85777c5d6cd52ad168741c9566186415aa26 2.9s
=> => extracting sha256:9bc548096c181514aa1253966a330134d939496027f92f57ab376cd236eb280b 0.0s
=> => extracting sha256:1d87379b86b89fd3b8bb1621128f00c8f962756e6aaaed264ec38db733273543 0.8s
=> [2/5] COPY . /app 2.3s
=> [3/5] WORKDIR /app 0.2s
=> [4/5] RUN pip install -U datasette 26.9s
=> [5/5] RUN datasette inspect covid.db --inspect-file inspect-data.json 3.1s
=> exporting to image 1.2s
=> => exporting layers 1.2s
=> => writing image sha256:b5db0c205cd3454c21fbb00ecf6043f261540bcf91c2dfc36d418f1a23a75d7a 0.0s

Use 'docker scan' to run Snyk tests against images to find vulnerabilities and learn how to fix them
Traceback (most recent call last):
"main", mod_spec)
File "c:\users\grott\anaconda3\lib\runpy.py", line 85, in _run_code
exec(code, run_globals)
File "C:\Users\grott\Anaconda3\Scripts\datasette.exe__main__.py", line 7, in <module>
File "c:\users\grott\anaconda3\lib\site-packages\click\core.py", line 829, in call
return self.main(args, kwargs)
File "c:\users\grott\anaconda3\lib\site-packages\click\core.py", line 782, in main
rv = self.invoke(ctx)
File "c:\users\grott\anaconda3\lib\site-packages\click\core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "c:\users\grott\anaconda3\lib\site-packages\click\core.py", line 1066, in invoke
return ctx.invoke(self.callback, ctx.params)
File "c:\users\grott\anaconda3\lib\site-packages\click\core.py", line 610, in invoke
return callback(args, **kwargs)
File "c:\users\grott\anaconda3\lib\site-packages\datasette\cli.py", line 283, in package
call(args)
File "c:\users\grott\anaconda3\lib\contextlib.py", line 119, in exit
next(self.gen)
File "c:\users\grott\anaconda3\lib\site-packages\datasette\utils__init__.py", line 451, in temporary_docker_directory
tmp.cleanup()
File "c:\users\grott\anaconda3\lib\tempfile.py", line 811, in cleanup
_shutil.rmtree(self.name)
File "c:\users\grott\anaconda3\lib\shutil.py", line 516, in rmtree
return _rmtree_unsafe(path, onerror)
File "c:\users\grott\anaconda3\lib\shutil.py", line 395, in _rmtree_unsafe
_rmtree_unsafe(fullname, onerror)
File "c:\users\grott\anaconda3\lib\shutil.py", line 404, in _rmtree_unsafe
onerror(os.rmdir, path, sys.exc_info())
File "c:\users\grott\anaconda3\lib\shutil.py", line 402, in _rmtree_unsafe
os.rmdir(path)
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\Users\grott\AppData\Local\Temp\tmpkb27qid3\datasette'```

Adds a method for retrieving all the teams within an organisation and all the members in those teams. The latter is stored as a join table team_members beteween teams and users.

a quick first pass at implementing the feature requested in https://github.com/dogsheep/github-to-sqlite/issues/64

testing instructions:

$ github-to-sqlite events events.db user/khimaros

if the specified user is the authenticated user, it will also include private events.

caveat: pagination appears to be broken (i don't see next in the response JSON from GitHub)

Description

Proposing a solution to the API limitation for merged_by in pull_requests. Specifically the following called out in the readme:

Note that the merged_by column on the pull_requests table will only be populated for pull requests that are loaded using the --pull-request option - the GitHub API does not return this field for pull requests that are loaded in bulk.

This approach might cause larger repos to hit rate limits called out in https://github.com/dogsheep/github-to-sqlite/issues/51 but seems to work well in the repos I tested and included below.

Old Behavior

• Had to list out the pull-requests individually via multiple --pull-request flags

New Behavior

• --merged-by flag for getting 'merge_by' information out of pull-requests without having to specify individual PR numbers.

Testing

Picking some repo that has more than one merger (datasette only has 1 😉 )

$ github-to-sqlite pull-requests ./github.db opnsense/tools --merged-by
$ echo "select id, url, merged_by from pull_requests;" | sqlite3 ./github.db 
83533612|https://github.com/opnsense/tools/pull/39|1915288
102632885|https://github.com/opnsense/tools/pull/43|1915288
149114810|https://github.com/opnsense/tools/pull/57|1915288
160394495|https://github.com/opnsense/tools/pull/64|1915288
163308408|https://github.com/opnsense/tools/pull/67|1915288
169723264|https://github.com/opnsense/tools/pull/69|1915288
171381422|https://github.com/opnsense/tools/pull/72|1915288
179938195|https://github.com/opnsense/tools/pull/77|1915288
196233824|https://github.com/opnsense/tools/pull/82|1915288
215289964|https://github.com/opnsense/tools/pull/93|
219696100|https://github.com/opnsense/tools/pull/97|1915288
223664843|https://github.com/opnsense/tools/pull/99|
228446172|https://github.com/opnsense/tools/pull/103|1915288
238930434|https://github.com/opnsense/tools/pull/110|1915288
255507110|https://github.com/opnsense/tools/pull/119|1915288
255980675|https://github.com/opnsense/tools/pull/120|1915288
261906770|https://github.com/opnsense/tools/pull/125|
263800503|https://github.com/opnsense/tools/pull/127|1915288
264038685|https://github.com/opnsense/tools/pull/128|1915288
264696704|https://github.com/opnsense/tools/pull/129|1915288
266660547|https://github.com/opnsense/tools/pull/130|1915288
273120409|https://github.com/opnsense/tools/pull/133|1915288
274370803|https://github.com/opnsense/tools/pull/135|
276600629|https://github.com/opnsense/tools/pull/139|
277303655|https://github.com/opnsense/tools/pull/141|1915288
293033714|https://github.com/opnsense/tools/pull/145|
294827649|https://github.com/opnsense/tools/pull/146|
295140008|https://github.com/opnsense/tools/pull/147|1915288
305690829|https://github.com/opnsense/tools/pull/150|9783985
307077931|https://github.com/opnsense/tools/pull/152|1915288
321782100|https://github.com/opnsense/tools/pull/155|
337265672|https://github.com/opnsense/tools/pull/160|
337267484|https://github.com/opnsense/tools/pull/161|1915288
368251763|https://github.com/opnsense/tools/pull/169|
428262505|https://github.com/opnsense/tools/pull/181|
437557011|https://github.com/opnsense/tools/pull/182|1915288
447079893|https://github.com/opnsense/tools/pull/185|
461822092|https://github.com/opnsense/tools/pull/191|
463290142|https://github.com/opnsense/tools/pull/193|1915288
470112962|https://github.com/opnsense/tools/pull/194|1915288
472644649|https://github.com/opnsense/tools/pull/195|1915288
488696898|https://github.com/opnsense/tools/pull/198|
513289902|https://github.com/opnsense/tools/pull/201|
522530265|https://github.com/opnsense/tools/pull/203|
564443347|https://github.com/opnsense/tools/pull/213|
597579516|https://github.com/opnsense/tools/pull/220|1915288
602860357|https://github.com/opnsense/tools/pull/221|1915288
608744738|https://github.com/opnsense/tools/pull/222|1915288
623279673|https://github.com/opnsense/tools/pull/228|1915288
664656182|https://github.com/opnsense/tools/pull/233|
664781786|https://github.com/opnsense/tools/pull/234|1915288
670683636|https://github.com/opnsense/tools/pull/235|1915288
683150764|https://github.com/opnsense/tools/pull/237|
685016233|https://github.com/opnsense/tools/pull/238|
687099825|https://github.com/opnsense/tools/pull/239|1915288
715705652|https://github.com/opnsense/tools/pull/244|1915288
715721248|https://github.com/opnsense/tools/pull/245|1915288

userid are now present for those PRs that were merged.

Without the flag the merged_by behavior remains missing as expected when get PRs bulk:

$ github-to-sqlite pull-requests ./github.db opnsense/tools
$ echo "select id, url, merged_by from pull_requests;" | sqlite3 ./github.db 
83533612|https://github.com/opnsense/tools/pull/39|
102632885|https://github.com/opnsense/tools/pull/43|
149114810|https://github.com/opnsense/tools/pull/57|
160394495|https://github.com/opnsense/tools/pull/64|
163308408|https://github.com/opnsense/tools/pull/67|
169723264|https://github.com/opnsense/tools/pull/69|
171381422|https://github.com/opnsense/tools/pull/72|
179938195|https://github.com/opnsense/tools/pull/77|
196233824|https://github.com/opnsense/tools/pull/82|
215289964|https://github.com/opnsense/tools/pull/93|
219696100|https://github.com/opnsense/tools/pull/97|
223664843|https://github.com/opnsense/tools/pull/99|
228446172|https://github.com/opnsense/tools/pull/103|
238930434|https://github.com/opnsense/tools/pull/110|
255507110|https://github.com/opnsense/tools/pull/119|
255980675|https://github.com/opnsense/tools/pull/120|
261906770|https://github.com/opnsense/tools/pull/125|
263800503|https://github.com/opnsense/tools/pull/127|
264038685|https://github.com/opnsense/tools/pull/128|
264696704|https://github.com/opnsense/tools/pull/129|
266660547|https://github.com/opnsense/tools/pull/130|
273120409|https://github.com/opnsense/tools/pull/133|
274370803|https://github.com/opnsense/tools/pull/135|
276600629|https://github.com/opnsense/tools/pull/139|
277303655|https://github.com/opnsense/tools/pull/141|
293033714|https://github.com/opnsense/tools/pull/145|
294827649|https://github.com/opnsense/tools/pull/146|
295140008|https://github.com/opnsense/tools/pull/147|
305690829|https://github.com/opnsense/tools/pull/150|
307077931|https://github.com/opnsense/tools/pull/152|
321782100|https://github.com/opnsense/tools/pull/155|
337265672|https://github.com/opnsense/tools/pull/160|
337267484|https://github.com/opnsense/tools/pull/161|
368251763|https://github.com/opnsense/tools/pull/169|
428262505|https://github.com/opnsense/tools/pull/181|
437557011|https://github.com/opnsense/tools/pull/182|
447079893|https://github.com/opnsense/tools/pull/185|
461822092|https://github.com/opnsense/tools/pull/191|
463290142|https://github.com/opnsense/tools/pull/193|
470112962|https://github.com/opnsense/tools/pull/194|
472644649|https://github.com/opnsense/tools/pull/195|
488696898|https://github.com/opnsense/tools/pull/198|
513289902|https://github.com/opnsense/tools/pull/201|
522530265|https://github.com/opnsense/tools/pull/203|
564443347|https://github.com/opnsense/tools/pull/213|
597579516|https://github.com/opnsense/tools/pull/220|
602860357|https://github.com/opnsense/tools/pull/221|
608744738|https://github.com/opnsense/tools/pull/222|
623279673|https://github.com/opnsense/tools/pull/228|
664656182|https://github.com/opnsense/tools/pull/233|
664781786|https://github.com/opnsense/tools/pull/234|
670683636|https://github.com/opnsense/tools/pull/235|
683150764|https://github.com/opnsense/tools/pull/237|
685016233|https://github.com/opnsense/tools/pull/238|
687099825|https://github.com/opnsense/tools/pull/239|
715705652|https://github.com/opnsense/tools/pull/244|
715721248|https://github.com/opnsense/tools/pull/245|

Individual PRs passed via --pull-request flag behaves as expected (unchanged):

$ github-to-sqlite pull-requests ./github.db opnsense/tools --pull-request 39 --pull-request 237
$ echo "select id, url, merged_by from pull_requests;" | sqlite3 ./github.db
83533612|https://github.com/opnsense/tools/pull/39|1915288
683150764|https://github.com/opnsense/tools/pull/237|

Picking 1 PR that has a merged_by (39) and one that does not (237)

I requested a Twitter export yesterday, and unfortunately they seem to have changed it such that twitter-to-sqlite import can't handle it anymore 😢

So far I've ran into two issues. The first was easy to work around, but the second will take more investigation. If I can find the time I'll keep working on it and update this issue accordingly.

The issues (so far):

1. Data seems to have moved to a data/ subdirectory

Running twitter-to-sqlite import on the raw zip file reports a bunch of "not yet implemented" errors, and then exits without actually importing anything:

❯ twitter-to-sqlite import tarchive.db twitter.zip
...
data/manifest: not yet implemented
data/account-creation-ip: not yet implemented
data/account-suspension: not yet implemented
... (dozens of more lines like this, including critical stuff like data/tweets) ...

(tarchive.db now exists, but is empty)

Workaround: unpack the zip file, and run twitter-to-sqlite import tarchive.db path/to/archive/data

That gets further, but:

2. Some schema(s?) have changed

At least, the blocks schema seems different now:

❯ twitter-to-sqlite import tarchive.db archive/data
direct-messages-group: not yet implemented
branch-links: not yet implemented
periscope-expired-broadcasts: not yet implemented
direct-messages: not yet implemented
mute: not yet implemented
Traceback (most recent call last):
  File "/Users/jacob/Library/Caches/pypoetry/virtualenvs/jacobian-dogsheep-4AXaN4tu-py3.8/bin/twitter-to-sqlite", line 8, in <module>
    sys.exit(cli())
  File "/Users/jacob/Library/Caches/pypoetry/virtualenvs/jacobian-dogsheep-4AXaN4tu-py3.8/lib/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/Users/jacob/Library/Caches/pypoetry/virtualenvs/jacobian-dogsheep-4AXaN4tu-py3.8/lib/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/Users/jacob/Library/Caches/pypoetry/virtualenvs/jacobian-dogsheep-4AXaN4tu-py3.8/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/jacob/Library/Caches/pypoetry/virtualenvs/jacobian-dogsheep-4AXaN4tu-py3.8/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/jacob/Library/Caches/pypoetry/virtualenvs/jacobian-dogsheep-4AXaN4tu-py3.8/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/Users/jacob/Library/Caches/pypoetry/virtualenvs/jacobian-dogsheep-4AXaN4tu-py3.8/lib/python3.8/site-packages/twitter_to_sqlite/cli.py", line 772, in import_
    archive.import_from_file(db, filepath.name, open(filepath, "rb").read())
  File "/Users/jacob/Library/Caches/pypoetry/virtualenvs/jacobian-dogsheep-4AXaN4tu-py3.8/lib/python3.8/site-packages/twitter_to_sqlite/archive.py", line 215, in import_from_file
    to_insert = transformer(data)
  File "/Users/jacob/Library/Caches/pypoetry/virtualenvs/jacobian-dogsheep-4AXaN4tu-py3.8/lib/python3.8/site-packages/twitter_to_sqlite/archive.py", line 115, in lists_member
    return {"lists-member": _list_from_common(data)}
  File "/Users/jacob/Library/Caches/pypoetry/virtualenvs/jacobian-dogsheep-4AXaN4tu-py3.8/lib/python3.8/site-packages/twitter_to_sqlite/archive.py", line 200, in _list_from_common
    for url in block["userListInfo"]["urls"]:
KeyError: 'urls'

That's as far as I got before I needed to work on something else. I'll report back if I get further!

My data already has an ID that comes from a federal agency.
Would love to have documentation on how to modify the template to:
- Remove the generated ID from the table
- Link the federal ID to the detail page
- and to ensure that the JSON file uses that as the ID. I'd be happy to include the database ID in the export, but not as a key.

I don't want to remove the ID from the database, though, because my experience with the federal agency is that data often has anomalies. I don't want all hell to break loose if they end up applying the same ID to multiple rows (which they haven't done yet). I just don't want it to display in the table or the data exports.

Perhaps this isn't a template issue, maybe more of a db manipulation...

Margie

I made a plugin for this today but I think perhaps it should be a default thing instead: https://datasette.io/plugins/datasette-template-request

I think it's best to still keep [ and ] out of column names though. Transforming them into ( and ) seems reasonable - but should that happen here or in sqlite-utils? I think in sqlite-utils.

_Originally posted by @simonw in https://github.com/simonw/datasette-app/issues/121#issuecomment-926200398_

This is a rethinking of the solution to:

• https://github.com/simonw/sqlite-utils/issues/86

When I use multiple files with the same name, e.g. in sqlite-utils memory a/bug.csv b/bug.csv, sqlite-utils creates invalid views.

Traceback (most recent call last):
  File "/home/karl/.local/bin/sqlite-utils", line 8, in <module>
    sys.exit(cli())
  File "/home/karl/.local/pipx/venvs/sqlite-utils/lib/python3.9/site-packages/click/core.py", line 1137, in __call__
    return self.main(*args, **kwargs)
  File "/home/karl/.local/pipx/venvs/sqlite-utils/lib/python3.9/site-packages/click/core.py", line 1062, in main
    rv = self.invoke(ctx)
  File "/home/karl/.local/pipx/venvs/sqlite-utils/lib/python3.9/site-packages/click/core.py", line 1668, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/karl/.local/pipx/venvs/sqlite-utils/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/karl/.local/pipx/venvs/sqlite-utils/lib/python3.9/site-packages/click/core.py", line 763, in invoke
    return __callback(*args, **kwargs)
  File "/home/karl/.local/pipx/venvs/sqlite-utils/lib/python3.9/site-packages/sqlite_utils/cli.py", line 1299, in memory
    db[csv_table].transform(types=tracker.types)
  File "/home/karl/.local/pipx/venvs/sqlite-utils/lib/python3.9/site-packages/sqlite_utils/db.py", line 1287, in transform
    self.db.execute(sql)
  File "/home/karl/.local/pipx/venvs/sqlite-utils/lib/python3.9/site-packages/sqlite_utils/db.py", line 421, in execute
    return self.conn.execute(sql)
sqlite3.OperationalError: error in view t1: no such table: main.bug

This can be reproduced with

#!/bin/bash
mkdir foo
mkdir bar
echo -e 'col1,col2\nval1,val2' > foo/bug.csv
echo -e 'col3,col4\nval3,val4' > bar/bug.csv
sqlite-utils memory */bug.csv 'SELECT 1'

Ideally, the tables would get unique names by including the next path segment until the names are unique. But just making the numbered t* aliases work would be good enough.

This problem can of course be worked around by renaming the files, but it would be nice if this case was handled more gracefully.

Thanks a lot for this great tool!

sqlite-utils query generates a JSON output with the result from the query:

[{...},{...}]

If no rows are returned by the query, I'm expecting an empty JSON array:

[]

But actually I'm getting an empty string. To be consistent, the output should be [] when the request succeeds (return code == 0).

To give an indication as to when the data was last updated.

This should be a field in the metadata that is then shown on the index page and in the footer, if it is set.

Also support setting it using an option to “datasette publish” and “datasette package” - which can either be a string or can be the magic string “today” to set it to today’s date:

datasette publish file.db --updated=today

Hi Simon,

It appears the fix for #57 doesn't fix things for me:

$ twitter-to-sqlite --version
twitter-to-sqlite, version 0.21.4
$ python --version
Python 3.9.6

$ twitter-to-sqlite home-timeline -a twitter-auth.json twitter/timeline.db --since
Importing tweets
Error: Use either --since or --since_id, not both

Is there any way I can help debug this?

Fixes remaining instances of this bug

Currently the _through clause can only work if the FK relationship is defined in a specific direction. I don't think there is any reason for this limitation, as an FK allows joining in both directions.

This is an admittedly hacky change to implement bidirectional joins using _through. It does work for our use-case, but I don't know if there are other implications that I haven't thought of. Also if this change is desirable we probably want to make the code a little nicer.

I have two connected sites:
http://www.SaferOrToxic.org
(a Hugo website)
and:
http://disinfectants.SaferOrToxic.org/disinfectants/listN
(a datasette table page)

The latter is linked as "The List" in the former's menu.
(I'd love a prettier URL, but that's what I've got.)

On:
http://disinfectants.SaferOrToxic.org/disinfectants/listN
... all the other menu links should point back to:
https://www.SaferOrToxic.org
And they do!

But the logo, for some reason--though it has an href pointing to:
https://www.SaferOrToxic.org
Keeps going to this instead:
https://disinfectants.saferortoxic.org/disinfectants/undefined

What is causing that? How can I fix it?

In #1284 back in March, I was doing battle with the index.html template, in a still unresolved issue. (I wanted only a single table page at the root.)

But I thought, well, if I can't resolve that, at least I could just point the main website to the datasette page ("The List,") and then have the List point back to the home website.

The menu hrefs to https://www.SaferOrToxic.org work just fine, exactly as they should, from the datasette page. Even the Home link works properly.

But the logo link keeps rewriting to: https://disinfectants.saferortoxic.org/disinfectants/undefined

This is the HTML:

<a class="text-3xl font-bold leading-none" href="https://www.saferortoxic.org"><img src="https://www.saferortoxic.org/images/logo_hu26e4dce8d5931af1ea33526b28fc8383_9734_c52a4f1635ef88bda858373270551ed2.webp" class="custom-logo" alt="Logo: Safer or Toxic?" width="300px"></a>

Is this somehow related to cloudflare?
Or something in the datasette code?

I'm starting to think it's a cloudflare issue.
https://user-images.githubusercontent.com/192568/133872043-5a008e14-88cb-43fd-b921-e63ff50636fb.png">

Can I at least rule out it being a datasette issue?

My repository is here:
https://github.com/mroswell/list-N

(BTW, I couldn't figure out how to reference a local image, either, on the datasette side, which is why I'm using the image from the www home page.)

Refs #217 - create a publish plugin that can deploy to AWS Lambda.

https://docs.aws.amazon.com/lambda/latest/dg/limits.html says lambda packages can be up to 50 MB, so this would only work with smaller databases (the command can check the filesize before attempting to package and deploy it).

Lambdas do get a 512 MB /tmp directory too, so for larger databases the function could start and then download up to 512MB from an S3 bucket - so the plugin could take an optional S3 bucket to write to and know how to upload the .db file there and then have the lambda download it on startup.

Bumps black from 21.7b0 to 21.8b0.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

I followed the instructions here, and even after running python update-docs-help.py I get the following failed tests -- any thoughts?

FAILED tests/test_api.py::test_searchable[/fixtures/searchable.json?_search=te*+AND+do*&_searchmode=raw-expected_rows3]
FAILED tests/test_api.py::test_searchmode[table_metadata1-_search=te*+AND+do*-expected_rows1]
FAILED tests/test_api.py::test_searchmode[table_metadata2-_search=te*+AND+do*&_searchmode=raw-expected_rows2]

This is with python 3.9.7 and lots of other packages, as in attached environment listing from conda list.
conda-installed.txt

This fixes documentation inaccuracies and adds a disclaimer about the signature of the get_metadata hook.

Addresses the following comments:
- https://github.com/simonw/datasette/issues/1384#issuecomment-869069926
- https://github.com/simonw/datasette/issues/1384#issuecomment-869075368

See https://datasette.io/desktop and https://simonwillison.net/2021/Sep/8/datasette-desktop/

Facets are awesome. But not when I need to join to tidy tables together. Or even just running explicitly the default SQL query that simply lists all the rows and columns of a table (up to SIZE). That is to say, when I browse a table, I see facets:

https://latest.datasette.io/fixtures/compound_three_primary_keys

But when I run a custom query, I don't:

https://latest.datasette.io/fixtures?sql=select+pk1%2C+pk2%2C+pk3%2C+content+from+compound_three_primary_keys+order+by+pk1%2C+pk2%2C+pk3+limit+101

Is there an idiom to cause custom SQL to come back with facet suggestions?

Fixes https://github.com/simonw/datasette/issues/1459

Adds support for --open --get /path to be used in combination.

If --open is provided alone, datasette will open a web page to a default URL.
If --get <url> is provided alone, datasette will output the result of doing a GET to that URL and then exit.
If --open --get <url> are provided together, datasette will open a web page to that URL.

TODO items:
- [ ] update documentation
- [ ] print out error message when --root --open --get <url> is used
- [ ] adjust code to require that <url> start with a / when -o --get <url> is used
- [ ] add test(s)

note, '@CTB' is used in this PR to flag code that needs revisiting.

I ditched "root" for "admin" because root by default gives you a whole bunch of stuff which I think could be confusing:

https://user-images.githubusercontent.com/9599/132412648-b768495b-ead2-4abe-875a-fd090fe3d71d.png">

Maybe the real problem here is that I'm conflating "root" permissions with "debug" options. Perhaps there should be an extra Datasette mode that unlocks debug tools for the root user?

_Originally posted by @simonw in https://github.com/simonw/datasette-app-support/issues/8#issuecomment-914638998_

https://github.com/asottile/blacken-docs

I have a table from the PUDL project (https://github.com/catalyst-cooperative/pudl) that looks like this:

CREATE TABLE fuel_ferc1 (
    id INTEGER NOT NULL, 
    record_id TEXT, 
    utility_id_ferc1 INTEGER, 
    report_year INTEGER, 
    plant_name_ferc1 TEXT, 
    fuel_type_code_pudl VARCHAR(7), 
    fuel_unit VARCHAR(7),
    fuel_qty_burned FLOAT,
    fuel_mmbtu_per_unit FLOAT, 
    fuel_cost_per_unit_burned FLOAT, 
    fuel_cost_per_unit_delivered FLOAT, 
    fuel_cost_per_mmbtu FLOAT, 
    PRIMARY KEY (id), 
    FOREIGN KEY(plant_name_ferc1, utility_id_ferc1) REFERENCES plants_ferc1 (plant_name_ferc1, utility_id_ferc1), 
    CONSTRAINT fuel_ferc1_fuel_type_code_pudl_enum CHECK (fuel_type_code_pudl IN ('coal', 'oil', 'gas', 'solar', 'wind', 'hydro', 'nuclear', 'waste', 'unknown')), 
    CONSTRAINT fuel_ferc1_fuel_unit_enum CHECK (fuel_unit IN ('ton', 'mcf', 'bbl', 'gal', 'kgal', 'gramsU', 'kgU', 'klbs', 'btu', 'mmbtu', 'mwdth', 'mwhth', 'unknown'))
);

Note that fuel_unit is a unit that pint can understand, and that fuel_qty_burned is a column of data that could be expressed in terms of actual units, not merely as a dimensionless number. Ditto the fuel_cost_per_unit_... columns. Is there a way to give a column a default metadata unit (such as tons or USD/ton) and then let that be overridden when the metadata in another column says barrels or USD/gramsU?

@catalyst-cooperative

(soft suggestion because I'm not sure I'm using datasette right yet)

Over at https://github.com/ctb/2021-sourmash-datasette, I'm playing around with datasette, and I'm creating some static pages to send people to the right facets. There may well be better ways of achieving this end goal, and I will find out if so, I'm sure!

But regardless I think it might be neat to support an option to allow -o/--open to take a relative URL, that then gets appended to the hostname and port. This would let me improve my documentation. I don't see any downsides, either, but 🤷 there may well be some :)

Happy to dig in and provide a PR if it's of interest. I'm not sure off the top of my head how to support an optional value to a parameter in argparse - the current -o behavior is kinda nice so it'd be suboptimal to require a url for -o. Maybe --open-url= or something would work?

Over in https://github.com/ctb/2021-sourmash-datasette, I discovered that the following commands fail:

conda create -n datasette4 -y datasette=0.58.1
conda activate datasette4
datasette gathertax.db

with ImportError: cannot import name 'WebSocketScope' from 'asgiref.typing'.

This appears to be because asgiref 3.3.4 doesn't have WebSocketScope, but later versions do - a simple

pip install asgiref==3.4.1

fixes the problem for me, at least to the point where I can run datasette and poke around as usual.

I note that over in the conda-forge recipe, https://github.com/conda-forge/datasette-feedstock/blob/master/recipe/meta.yaml pins asgiref to < 3.4.0, but I'm not sure why - so I'm not sure how to best resolve this issue :).

Not sure if you want them mentioned explicitly (it's already a long list), but following up on https://twitter.com/simonw/status/1434176989565382656

While working on updating sqlite-utils for NixOS/Nixpkgs, I came a cross the following:

In 5ec6686153e29ae10d4921a1ad4c841f192f20e2, a new dependency was added on dateutils (https://pypi.org/project/dateutils/).

I believe this is unintentional, and instead python-dateutil (https://pypi.org/project/python-dateutil/) was intended.

My reasoning is:
- python-dateutil is imported here in recipes.py
- The mypy type-python-dateutil dependency in setup.py
- python-dateutil is a dependency of dateutils as seen in the output in docs/tutorial.ipynb

Seems like the trailing "s" seems to be the source of confusion 😅

I've swapped the dependencies out, hope this helps.

This makes it so you can use Dogsheep if you're using Datasette with the --base-url /some-path/ setting.

This is an error message I get upon parsing the enex file of my Inbox. Please find the full error message below. Any hints welcome.

Importing from ENEX  [##################------------------]   50%  00:00:50
Traceback (most recent call last):
  File "/Users/utopist/.virtualenvs/evernote-to-sqlite-Og2PIW3Y/bin/evernote-to-sqlite", line 8, in <module>
    sys.exit(cli())
  File "/Users/utopist/.virtualenvs/evernote-to-sqlite-Og2PIW3Y/lib/python3.9/site-packages/click/core.py", line 1137, in __call__
    return self.main(*args, **kwargs)
  File "/Users/utopist/.virtualenvs/evernote-to-sqlite-Og2PIW3Y/lib/python3.9/site-packages/click/core.py", line 1062, in main
    rv = self.invoke(ctx)
  File "/Users/utopist/.virtualenvs/evernote-to-sqlite-Og2PIW3Y/lib/python3.9/site-packages/click/core.py", line 1668, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/utopist/.virtualenvs/evernote-to-sqlite-Og2PIW3Y/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/utopist/.virtualenvs/evernote-to-sqlite-Og2PIW3Y/lib/python3.9/site-packages/click/core.py", line 763, in invoke
    return __callback(*args, **kwargs)
  File "/Users/utopist/.virtualenvs/evernote-to-sqlite-Og2PIW3Y/lib/python3.9/site-packages/evernote_to_sqlite/cli.py", line 30, in enex
    for tag, note in find_all_tags(fp, ["note"], progress_callback=bar.update):
  File "/Users/utopist/.virtualenvs/evernote-to-sqlite-Og2PIW3Y/lib/python3.9/site-packages/evernote_to_sqlite/utils.py", line 17, in find_all_tags
    for event, el in parser.read_events():
  File "/usr/local/Cellar/python@3.9/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/xml/etree/ElementTree.py", line 1329, in read_events
    raise event
  File "/usr/local/Cellar/python@3.9/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/xml/etree/ElementTree.py", line 1301, in feed
    self._parser.feed(data)
xml.etree.ElementTree.ParseError: mismatched tag: line 6837961, column 2

The neatest way to have the footer stick to the bottom of the browser window that I've found is to use the flexbox pattern from https://css-tricks.com/couple-takes-sticky-footer/

<body>
  <div class="content">
    content
  </div>
  <footer class="footer"></footer>
</body>

html, body {
  height: 100%;
}
body {
  display: flex;
  flex-direction: column;
}
.content {
  flex: 1 0 auto;
}
.footer {
  flex-shrink: 0;
}

I tried this in a custom plugin but it ended up having to duplicate the entire base.html template just to get a wrapper around the not-footer content.

I think Datasette's own base.html template should have this wrapper element instead.

Hi,

First of all, thank you for this wonderful project :smile:

I started to use dogsheep to make my personal data searchable, and by using the project I noticed an issue with the index command.

It always expects you are running it from the root folder from where the data is located, so I got some errors while trying to make it work on my setup.

I separate all databases inside a data folder (I published my setup to be easier to follow: https://github.com/humrochagf/my-dogsheep)

Before, I configured dogsheep.yml to add the data folder to its path like this:

data/twitter.db:
    tweets:
        sql: |-
...

And running the index command like this:

dogsheep-beta index data/dogsheep.db dogsheep.yml

It worked to the normal search feature with no problem this way, but when I started adding display_sql rules the app started to crash, because at datasette get_database it was looking for data/twitter and it only had a db called twitter there.

So my workaround to that was to cd into the data folder and run the indexer. You can check the way I'm doing it at this line of the makefile: https://github.com/humrochagf/my-dogsheep/blob/main/makefile#L3

It works but it would be nice to have an option to pass the path where the data is located to the index function.

Hello,

I am a huge fan of this being used for exploring data. I think it has a lot of flexibility not found in other tools.

I'm not sure if what I'm asking for is possible: Can this be extended to publish to IPFS?

IPFS is an attractive hosting option for decentralized journalism.

Food for thought ~

The datasette CLI tool currently has 7 subcommands: serve, inspect, install, package, plugins, publish and uninstall.

A plugin hook could allow plugins to register extra subcommands.

I've avoided this for quite a while because I didn't have good use-cases for it - but the existence of the datasette install xxx command for installing packages into the correct virtual environment means that actually there's a good reason to do this: it would allow plugins to provide additional command-line mechanisms without the user having to understand how virtual environments work in order to install those commands into the same environment as the rest of Datasette.

Workflows that have an id in any step result in the following error when running workflows:

e.g.github-to-sqlite workflows github.db nixos/nixpkgs

Traceback (most recent call last): File "/usr/local/bin/github-to-sqlite", line 8, in <module> sys.exit(cli()) File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1137, in __call__ return self.main(*args, **kwargs) File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1062, in main rv = self.invoke(ctx) File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1668, in invokeTraceback (most recent call last):
File "/usr/local/bin/github-to-sqlite", line 8, in <module>
sys.exit(cli())
File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1137, in call
return self.main(args, kwargs)
File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1062, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1668, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1404, in invoke
return ctx.invoke(self.callback, ctx.params)
File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 763, in invoke
return __callback(args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/github_to_sqlite/cli.py", line 601, in workflows
utils.save_workflow(db, repo_id, filename, content)
File "/usr/local/lib/python3.8/dist-packages/github_to_sqlite/utils.py", line 865, in save_workflow
db["steps"].insert_all(
File "/usr/local/lib/python3.8/dist-packages/sqlite_utils/db.py", line 2596, in insert_all
self.insert_chunk(
File "/usr/local/lib/python3.8/dist-packages/sqlite_utils/db.py", line 2378, in insert_chunk
result = self.db.execute(query, params)
File "/usr/local/lib/python3.8/dist-packages/sqlite_utils/db.py", line 419, in execute
return self.conn.execute(sql, parameters)
sqlite3.IntegrityError: datatype mismatch
```

• Information about the ID key in a step for GHA
• An example workflow from a public repo

Changes

I'm proposing that the key for id in step is replaced with step_id so that it no longer interferes with the table id for tracking the record.

Special thanks to @sarcasticadmin @egiffen and @ruebenramirez for helping a bit on this 😄

https://github.com/simonw/datasette/blob/a1a33bb5822214be1cebd98cd858b2058d91a4aa/datasette/cli.py#L122-L127

_Originally spotted by @simonw in https://github.com/simonw/datasette/issues/1449#issuecomment-907539668_

~ % datasette --help
Usage: datasette [OPTIONS] COMMAND [ARGS]...

  Datasette!

Options:
  --version  Show the version and exit.
  --help     Show this message and exit.

the library deal with Postgres type jsonb as dictionary, add dict type as a TEXT for mapping to sqlite

Like on https://til.simonwillison.net/til/search?q=Snippet

Got this error today:

(evernote-to-sqlite) /tmp % evernote-to-sqlite enex evernote.db simonwillison\'s\ notebook.enex 
Importing from ENEX  [######------------------------------]   17%
Traceback (most recent call last):
  File "/Users/simon/.local/bin/evernote-to-sqlite", line 8, in <module>
    sys.exit(cli())
  File "/Users/simon/.local/pipx/venvs/evernote-to-sqlite/lib/python3.9/site-packages/click/core.py", line 1137, in __call__
    return self.main(*args, **kwargs)
  File "/Users/simon/.local/pipx/venvs/evernote-to-sqlite/lib/python3.9/site-packages/click/core.py", line 1062, in main
    rv = self.invoke(ctx)
  File "/Users/simon/.local/pipx/venvs/evernote-to-sqlite/lib/python3.9/site-packages/click/core.py", line 1668, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/simon/.local/pipx/venvs/evernote-to-sqlite/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/simon/.local/pipx/venvs/evernote-to-sqlite/lib/python3.9/site-packages/click/core.py", line 763, in invoke
    return __callback(*args, **kwargs)
  File "/Users/simon/.local/pipx/venvs/evernote-to-sqlite/lib/python3.9/site-packages/evernote_to_sqlite/cli.py", line 31, in enex
    save_note(db, note)
  File "/Users/simon/.local/pipx/venvs/evernote-to-sqlite/lib/python3.9/site-packages/evernote_to_sqlite/utils.py", line 36, in save_note
    content = ET.tostring(ET.fromstring(content_xml)).decode("utf-8")
  File "/usr/local/Cellar/python@3.9/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/xml/etree/ElementTree.py", line 1347, in XML
    parser.feed(text)
xml.etree.ElementTree.ParseError: not well-formed (invalid token): line 2, column 132

To spot duplicate images (where the file content differs such that the sha256 is no longer a match) it would be useful to calculate and store perceptual hashes of some sort.

Hey,
I have a database that I save in HTML from couple of web scrapers. There are around 200k+, 50+ rows in a couple of tables, with sqlite file weighing around 600MB.

The app runs on a VPS with 2 core CPU, 4GB RAM and refreshing database page regularly takes more than 10 seconds. I was suspecting that counting tables was the culprit, but manually running select count(*) from table_name for the largest table finishes under a second.

I've looked at the source code. There's a check for index page for mutable databases larger than 100MB
https://github.com/simonw/datasette/blob/799c5d53570d773203527f19530cf772dc2eeb24/datasette/views/index.py#L15

but this check is not performed for database page.
I've manually crippled Database::table_counts method

async def table_counts(self, limit=10):
    if not self.is_mutable and self.cached_table_counts is not None:
        return self.cached_table_counts
    # Try to get counts for each table, $limit timeout for each count
    counts = {}
    for table in await self.table_names():
        try:
            # table_count = (
            #     await self.execute(
            #         "select count(*) from [{}]".format(table),
            #         custom_time_limit=limit,
            #     )
            # ).rows[0][0]
            counts[table] = 10 # table_count
        # In some cases I saw "SQL Logic Error" here in addition to
        # QueryInterrupted - so we catch that too:
        except (QueryInterrupted, sqlite3.OperationalError, sqlite3.DatabaseError):
            counts[table] = None
    if not self.is_mutable:
        self.cached_table_counts = counts
    return counts

now the page loads in <100ms.

Is it possible to apply size check on database page too?

{
    "python": {
        "version": "3.8.0",
        "full": "3.8.0 (default, Oct 28 2019, 16:14:01) \n[GCC 8.3.0]"
    },
    "datasette": {
        "version": "0.44"
    },
    "asgi": "3.0",
    "uvicorn": "0.11.5",
    "sqlite": {
        "version": "3.22.0",
        "fts_versions": [
            "FTS5",
            "FTS4",
            "FTS3"
        ],
        "extensions": {
            "json1": null
        },
        "compile_options": [
            "COMPILER=gcc-7.4.0",
            "ENABLE_COLUMN_METADATA",
            "ENABLE_DBSTAT_VTAB",
            "ENABLE_FTS3",
            "ENABLE_FTS3_PARENTHESIS",
            "ENABLE_FTS3_TOKENIZER",
            "ENABLE_FTS4",
            "ENABLE_FTS5",
            "ENABLE_JSON1",
            "ENABLE_LOAD_EXTENSION",
            "ENABLE_PREUPDATE_HOOK",
            "ENABLE_RTREE",
            "ENABLE_SESSION",
            "ENABLE_STMTVTAB",
            "ENABLE_UNLOCK_NOTIFY",
            "ENABLE_UPDATE_DELETE_LIMIT",
            "HAVE_ISNAN",
            "LIKE_DOESNT_MATCH_BLOBS",
            "MAX_SCHEMA_RETRY=25",
            "MAX_VARIABLE_NUMBER=250000",
            "OMIT_LOOKASIDE",
            "SECURE_DELETE",
            "SOUNDEX",
            "TEMP_STORE=1",
            "THREADSAFE=1"
        ]
    }
}

It currently works like this: https://github.com/simonw/sqlite-utils/blob/77c240df56068341561e95e4a412cbfa24dc5bc7/sqlite_utils/db.py#L2177-L2195

It's registering a function called convert_value() and then failing to de-register that function once it has finished.

It might even be possible for two queries running against the same connection to clobber each other's convert_value() functions, leading to incorrect behaviour.

So two fixes: firstly it should register the function with a unique name (maybe add a random suffix). Secondly, it should de-register that function once it has finished.

Bumps black from 21.6b0 to 21.7b0.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Updates the requirements on trustme to permit the latest version.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

The fulltext search documentation refers to the parameter as searchmode but the metadata.json example uses search_mode. The latter doesn't actually seem to work.

'insert-files' creates BLOB columns for file contents. Transforming the column to TEXT still keep the content as binary. Even though I'm sure there is a transform that can be applied decoding the text it would be great to have a argument to make 'insert-files' to do it as text (with optional text encoding).

The use case is a bunch of htmls (single file) on a directory structure that inserted with this command could be served in Datasette allowing full text search.

Refs #319.

This solves a weird issue when there is record with metadata key
that is only different in letter cases.

See the test for details.

This closes #21.

When there are metadata entries with the same case insensitive string, then there is an error when trying to create a new column for that metadata entry in the database table, because a column with that case insensitive name already exists.

 <Record type="HKQuantityTypeIdentifierDietaryCholesterol" sourceName="MyFitnessPal" sourceVersion="35120" unit="mg" creationDate="2021-07-04 20:55:27 +0200" startDate="2021-07-04 20:55:00 +0200" endDate="2021-07-04 20:55:00 +0200" value="124">
  <MetadataEntry key="meal" value="Dinner"/>
  <MetadataEntry key="Meal" value="Dinner"/>
 </Record>

The code added in this PR checks if a key already exists in a record and if so adds a number at its end. The resulting column names look like the example below then. Interestingly, the column names viewed with Datasette are not case insensitive.

startDate, endDate, value, unit, sourceName, sourceVersion, creationDate, metadata_meal, metadata_Meal_2, metadata_Mahlzeit

When I'm working with new data I often find myself wanting to run a search for text embedded in ANY of the columns of a table, without having to even fully understand the schema first.

I figured out a trick for doing that using a SQL-generated SQL query here: https://til.simonwillison.net/datasette/search-all-columns-trick

But maybe this should be a core Datasette feature? Or a plugin?

Hey, thank you for this repo!

When I try to convert my export, I get a multiple column error. Here is the stack trace:

(.venv) (base) computer:bodyweight_app user$ healthkit-to-sqlite ./data/Health_export.zip ./data/healthkit.db
Importing from HealthKit  [###############################-----]   87%  00:00:22
Traceback (most recent call last):
  File "/MyProject/.venv/bin/healthkit-to-sqlite", line 10, in <module>
    sys.exit(cli())
  File "/MyProject/.venv/lib/python3.7/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/MyProject/.venv/lib/python3.7/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/MyProject/.venv/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/MyProject/.venv/lib/python3.7/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/MyProject/.venv/lib/python3.7/site-packages/healthkit_to_sqlite/cli.py", line 57, in cli
    convert_xml_to_sqlite(fp, db, progress_callback=bar.update, zipfile=zf)
  File "/MyProject/.venv/lib/python3.7/site-packages/healthkit_to_sqlite/utils.py", line 41, in convert_xml_to_sqlite
    write_records(records, db)
  File "/MyProject/.venv/lib/python3.7/site-packages/healthkit_to_sqlite/utils.py", line 146, in write_records
    batch_size=50,
  File "/MyProject/.venv/lib/python3.7/site-packages/sqlite_utils/db.py", line 2579, in insert_all
    extracts=extracts,
  File "/MyProject/.venv/lib/python3.7/site-packages/sqlite_utils/db.py", line 1246, in create
    extracts=extracts,
  File "/MyProject/.venv/lib/python3.7/site-packages/sqlite_utils/db.py", line 767, in create_table
    self.execute(sql)
  File "/MyProject/.venv/lib/python3.7/site-packages/sqlite_utils/db.py", line 421, in execute
    return self.conn.execute(sql)
sqlite3.OperationalError: duplicate column name: metadata_Meal

To provide a way of running analyze-tables directly against JSON or CSV data.

If a text column actually contains numbers, being able to "sort by column, treated as numeric" would be really useful.

Probably depends on column actions enabled by #690

Sorting that by date makes sense for seeing most recent points, and my DB has 2.5m points in so it's an expensive sort!

Got this error today against a fresh export:

table activity_summary has no column named appleMoveTime

This fixes the issues discussed in #54

(I discovered this trawling the issues for how to use --since with favorites)

I'm using the following command:

twitter-to-sqlite user-timeline -a twitter-auth.json twitter/tweets.db --since

Which gives the following error:

Error: Use either --since or --since_id, not both

Running without --since.

Traceback (most recent call last):
  File "/usr/local/bin/twitter-to-sqlite", line 8, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1137, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1062, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1668, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 763, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/twitter_to_sqlite/cli.py", line 317, in user_timeline
    for tweet in bar:
  File "/usr/local/lib/python3.9/site-packages/click/_termui_impl.py", line 328, in generator
    for rv in self.iter:
  File "/usr/local/lib/python3.9/site-packages/twitter_to_sqlite/utils.py", line 234, in fetch_user_timeline
    yield from fetch_timeline(
  File "/usr/local/lib/python3.9/site-packages/twitter_to_sqlite/utils.py", line 202, in fetch_timeline
    raise Exception(str(tweets["errors"]))
Exception: [{'code': 44, 'message': 'since_id parameter is invalid.'}]

Python 3.9.5
twitter-to-sqlite, version 0.21.3

See accompanying Twitter thread: https://twitter.com/simonw/status/1424820203603431439

Datasette currently has a plugin for configuring robots.txt, but I'm beginning to think it should be part of core and crawlers should be blocked by default - having people explicitly opt-in to having their sites crawled and indexed feels a lot safer https://datasette.io/plugins/datasette-block-robots

I have a lot of Datasettes deployed now, and tailing logs shows that they are being hammered by search engine crawlers even though many of them are not interesting enough to warrant indexing.

I'm starting to think blocking crawlers would actually be a better default for most people, provided it was well documented and easy to understand how to allow them.

Default-deny is usually a better policy than default-allow!

https://github.com/simonw/datasette/blob/adb5b70de5cec3c3dd37184defe606a082c232cf/datasette/app.py#L231

I want to use it in https://github.com/simonw/datasette-block-robots/issues/5

Refs #1293, follows #942.

Thanks again for such a powerful project.

For deploying to cloudrun from github actions, I'd like to create a service account with minimal permissions.

It would be great to document what those minimum permission that need to be set in the IAM.

A useful capability would be if it was super-easy to say "any pushes to branch X should be deployed to latest-X.datasette.io".

I'd like to use this for the column query information work in #1434

See #1442.