id: 7B_nXLYazAyEryn4Z9Yz0
date: 2024-03-28T10:23:32.409Z
summary: Rewriting discord chat into docs
messages:
- id: 8cMmQ9e-iMP67kGcFHFkg
date: 2024-03-28T10:20:42.845Z
type: system
text: >
I am ChatCraft, a web-based, expert programming AI assistant. I help
programmers learn, experiment, and be more creative with code.
I follow these rules when responding:
- Use GitHub flavored Markdown
- ALWAYS include the programming language name (js) or type of data (csv) at the start of Markdown code blocks
- Format ALL lines of code to 80 characters or fewer
- Use Mermaid diagrams when discussing visual topics
- If using functions, only use the specific functions I have been provided with
- If responding with math markdown, inline or otherwise, I use KaTeX syntax in math Markdown by enclosing EVERY mathematical expression, equation, variable, and formula with double-dollar signs `($$)`, for example: $$O(n\log n)$$, $$1024 * 1024 = 1048576$$, $$1024^2$$, $$X$$
imageUrls: []
- id: hMU0DChEApBgcY0AXqys9
date: 2024-03-28T10:21:20.181Z
type: human
text: >-
```
taras: fyi
echo "SELECT * FROM vals WHERE lower(name) LIKE '%feed%' and lower(name) like '%email%' LIMIT 100" | jq -R '{args: [.]} ' | xargs -0 -I {} curl -X POST "https://sqlite-execute.web.val.run" -H "Content-Type: application/json" -d {} |yq -P
worked great
[6:28 PM]taras: is this a mirror of your postgres main db?
[1:11 AM]vladimyr: @taras Few notes on how you can improve this pipeline:
You don't need xargs for taking input from stdin, curl supports -d '@-' where @ means file and dash means stdin
Recent versions of curl support --json flag that automatically sets correct HTTP headers so you can simply use: --json '@-'
Nitpick but explicit setting of HTTP method in curl invocations is considered bad practice because it is not needed in case of data, form and/or json flags plus it has not so obvious side effects when more than one request is being done behind the scenes
Finally when working with JSON there is common pattern of using trinity of jo, curl and jq. While you can definitely (ab)use jq to produce json payload it is typically done using jo. In your example it would look something like:
$ jo args="$(jo -a "SELECT * FROM vals WHERE lower(name) LIKE '%feed%' and lower(name) like '%email%' LIMIT 100")"
[1:12 AM]vladimyr: Combine it all together and you get something like this:
$ jo args="$(jo -a "SELECT * FROM vals WHERE lower(name) LIKE '%feed%' and lower(name) like '%email%' LIMIT 100")" \
| curl -s --json '@-' 'https://sqlite-execute.web.val.run'
[1:15 AM]vladimyr: I highly recommend everyone to read through https://everything.curl.dev/http/post/json.html and Everything curl online book in general. It is full of useful tricks and will definitely elevate one's curl game one step further
JSON - everything curl
everything there is to know about curl, libcurl and the cURL project
[1:19 AM]vladimyr: Oh and for sake of keeping SQL statement as input to your pipeline:
$ echo "SELECT * FROM vals WHERE lower(name) LIKE '%feed%' and lower(name) like '%email%' LIMIT 100" \
| jo -a '@-' \
| jo args=':-' \
| curl -s --json '@-' 'https://sqlite-execute.web.val.run'
[1:20 AM]vladimyr: @- means treat stdin as string, :- means treat stdin as JSON
```
i'm taras, vlad totally schooled me.
Rewrite this into a markdown doc while retaining exact language and examples used whenever possible
imageUrls: []
user:
username: tarasglek
name: Taras Glek
avatarUrl: https://avatars.githubusercontent.com/u/857083?v=4
- id: rk9mHNh807JWHhWjaOYYF
date: 2024-03-28T10:21:20.403Z
type: ai
text: >-
# Improving Command Line Data Processing with `curl` and `jq`
In a recent discussion, a useful command line pipeline was shared for querying a database and processing the results. The original command looked like this:
```bash
echo "SELECT * FROM vals WHERE lower(name) LIKE '%feed%' and lower(name) like '%email%' LIMIT 100" | jq -R '{args: [.]} ' | xargs -0 -I {} curl -X POST "https://sqlite-execute.web.val.run" -H "Content-Type: application/json" -d {} | yq -P
```
This command effectively queries a database for records matching certain criteria and processes the results. However, there are several ways this pipeline can be improved for efficiency and clarity.
## Recommendations for Improvement
1. **Elimination of `xargs` for stdin input in `curl`:** Instead of using `xargs` to pass input from stdin, `curl` supports `-d '@-'` where `@` means file and `-` means stdin. This simplifies the command.
2. **Use of `--json` flag in `curl`:** Recent versions of `curl` support the `--json` flag, which automatically sets the correct HTTP headers for JSON content. This allows for a more straightforward command.
3. **Avoid explicit setting of HTTP method in `curl`:** Explicitly setting the HTTP method (e.g., `-X POST`) is considered bad practice when using data, form, and/or json flags. This is because it's unnecessary and can lead to unexpected side effects when more than one request is made behind the scenes.
4. **Use of `jo` for JSON payload creation:** While `jq` can be used to create a JSON payload, it's more common and cleaner to use `jo` for this purpose. `jo` is specifically designed for creating JSON objects and arrays.
### Improved Command Examples
Using the recommendations above, the command can be rewritten as follows:
```bash
$ jo args="$(jo -a "SELECT * FROM vals WHERE lower(name) LIKE '%feed%' and lower(name) like '%email%' LIMIT 100")" \
| curl -s --json '@-' 'https://sqlite-execute.web.val.run'
```
This command uses `jo` to create the JSON payload and then passes it to `curl` using the `--json` flag for processing.
For keeping the SQL statement as input to the pipeline, the command can be further refined:
```bash
$ echo "SELECT * FROM vals WHERE lower(name) LIKE '%feed%' and lower(name) like '%email%' LIMIT 100" \
| jo -a '@-' \
| jo args=':-' \
| curl -s --json '@-' 'https://sqlite-execute.web.val.run'
```
In this version, `@-` means treat stdin as a string, and `:-` means treat stdin as JSON, allowing for dynamic input directly into the JSON payload.
## Additional Resources
For those looking to deepen their understanding of `curl` and its capabilities, especially with JSON, it's highly recommended to read through [Everything curl](https://everything.curl.dev/http/post/json.html). This online book is full of useful tricks and insights that can significantly enhance one's command line data processing skills.
---
By incorporating these suggestions, the efficiency and readability of command line data processing pipelines can be greatly improved, making them more maintainable and easier to understand.
imageUrls: []
model: gpt-4-turbo-preview
versions:
- id: Q6Q2XL7CbThIYdTGrPTuc
date: 2024-03-28T10:21:43.993Z
model: gpt-4-turbo-preview
text: >-
# Improving Command Line Data Processing with `curl` and `jq`
In a recent discussion, a useful command line pipeline was shared for querying a database and processing the results. The original command looked like this:
```bash
echo "SELECT * FROM vals WHERE lower(name) LIKE '%feed%' and lower(name) like '%email%' LIMIT 100" | jq -R '{args: [.]} ' | xargs -0 -I {} curl -X POST "https://sqlite-execute.web.val.run" -H "Content-Type: application/json" -d {} | yq -P
```
This command effectively queries a database for records matching certain criteria and processes the results. However, there are several ways this pipeline can be improved for efficiency and clarity.
## Recommendations for Improvement
1. **Elimination of `xargs` for stdin input in `curl`:** Instead of using `xargs` to pass input from stdin, `curl` supports `-d '@-'` where `@` means file and `-` means stdin. This simplifies the command.
2. **Use of `--json` flag in `curl`:** Recent versions of `curl` support the `--json` flag, which automatically sets the correct HTTP headers for JSON content. This allows for a more straightforward command.
3. **Avoid explicit setting of HTTP method in `curl`:** Explicitly setting the HTTP method (e.g., `-X POST`) is considered bad practice when using data, form, and/or json flags. This is because it's unnecessary and can lead to unexpected side effects when more than one request is made behind the scenes.
4. **Use of `jo` for JSON payload creation:** While `jq` can be used to create a JSON payload, it's more common and cleaner to use `jo` for this purpose. `jo` is specifically designed for creating JSON objects and arrays.
### Improved Command Examples
Using the recommendations above, the command can be rewritten as follows:
```bash
$ jo args="$(jo -a "SELECT * FROM vals WHERE lower(name) LIKE '%feed%' and lower(name) like '%email%' LIMIT 100")" \
| curl -s --json '@-' 'https://sqlite-execute.web.val.run'
```
This command uses `jo` to create the JSON payload and then passes it to `curl` using the `--json` flag for processing.
For keeping the SQL statement as input to the pipeline, the command can be further refined:
```bash
$ echo "SELECT * FROM vals WHERE lower(name) LIKE '%feed%' and lower(name) like '%email%' LIMIT 100" \
| jo -a '@-' \
| jo args=':-' \
| curl -s --json '@-' 'https://sqlite-execute.web.val.run'
```
In this version, `@-` means treat stdin as a string, and `:-` means treat stdin as JSON, allowing for dynamic input directly into the JSON payload.
## Additional Resources
For those looking to deepen their understanding of `curl` and its capabilities, especially with JSON, it's highly recommended to read through [Everything curl](https://everything.curl.dev/http/post/json.html). This online book is full of useful tricks and insights that can significantly enhance one's command line data processing skills.
---
By incorporating these suggestions, the efficiency and readability of command line data processing pipelines can be greatly improved, making them more maintainable and easier to understand.